nlp笔试题


最近面试一家公司nlp岗位,这是当时出的一道算法题,时间比较紧没答出来,请教各位大佬
题目:用户在输入文本时偶尔会出错,假设给你一本结构化的词典,里面有所有词和每个词在文章中出现的频率,怎么样实现一个纠错方法。
a.假设文本时英文单词组成,键盘敲错的概率为q,如何实现一个纠错方案?尝试用代码实现你的方案。(提示:贝叶斯定理)
b.当文本时中文时,你上面的方案是否合适,如果假设中文是使用拼音作为输入法,上面的方案需要做什么样的调整?如果想提高准确性还需要什么信息?
已邀请:

张楚岚

赞同来自:


import re
from collections import Counter

def words(text): return re.findall(r'\w ', text.lower())

WORDS = Counter(words(open('big.txt').read()))

def P(word, N=sum(WORDS.values())):
"Probability of word."
return WORDS[word] / N

def correction(word):
"Most probable spelling correction for word."
return max(candidates(word), key=P)

def candidates(word):
"Generate possible spelling corrections for word."
return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

def known(words):
"The subset of words that appear in the dictionary of WORDS."
return set(w for w in words if w in WORDS)

def edits1(word):
"All edits that are one edit away from word."
letters = 'abcdefghijklmnopqrstuvwxyz'
splits = [(word[:i], word[i:]) for i in range(len(word) 1)]
deletes = [L R[1:] for L, R in splits if R]
transposes = [L R[1] R[0] R[2:] for L, R in splits if len(R)>1]
replaces = [L c R[1:] for L, R in splits if R for c in letters]
inserts = [L c R for L, R in splits for c in letters]
return set(deletes transposes replaces inserts)

def edits2(word):
"All edits that are two edits away from word."
return (e2 for e1 in edits1(word) for e2 in edits1(e1))

这是在网上找到的一个拼写纠错的,供大家参考,链接:http://norvig.com/spell-correct.html

要回复问题请先登录注册

返回顶部