Go 语言高效分词, 支持英文、中文、日文等

发表于

Go efficient text segmentation; support english, chinese, japanese and other.

CircleCI Status codecov Build Status Go Report Card GoDoc Release Join the chat at https://gitter.im/go-ego/ego

简体中文

Dictionary with double array trie (Double-Array Trie) to achieve, Sender algorithm is the shortest path
based on word frequency plus dynamic programming.

Support common and search engine two participle mode, support user dictionary, POS tagging, run JSON RPC service.

Text Segmentation speed single thread 9MB/s，goroutines concurrent 42MB/s (8 nuclear Macbook
Pro).

Install / update

1
复制代码`go get -u github.com/go-ego/gse`

Build-tools

1
复制代码`go get -u github.com/go-ego/re`

re gse

To create a new gse application

1
复制代码`$ re gse my-gse`

re run

To run the application we just created, you can navigate to the application folder and execute:

1
复制代码`$ cd my-gse && re run`

Use

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
复制代码package main

import (
	"fmt"

	"github.com/go-ego/gse"
)

func main() {
	// Load the dictionary
	var segmenter gse.Segmenter
	segmenter.LoadDict()
	// segmenter.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")

	// Text Segmentation
	text := []byte("你好世界, Hello world.")
	segments := segmenter.Segment(text)
  
	// Handle word segmentation results
	// Support for normal mode and search mode two participle, see the comments in the code ToString function.
	fmt.Println(gse.ToString(segments, false)) 
}

License

Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), base on sego

本文转载自: 掘金

开发者博客 – 和开发相关的 这里全都有

0%