Token Annotation¶
This annotation type introduces a tokenisation layer for the document. The terms token and word are used interchangeably in FoLiA as FoLiA itself does not commit to a specific tokenisation paradigm. Tokenisation is a prerequisite for the majority of linguistic annotation types offered by FoLiA and it is one of the most fundamental types of Structure Annotation. The words/tokens are typically embedded in other types of structure elements, such as sentences or paragraphs.
Specification¶
Annotation Category: | |
---|---|
Declaration: |
|
Version History: | |
Since the beginning |
|
Element: |
|
API Class: |
|
Required Attributes: | |
Optional Attributes: | |
|
|
Accepted Data: |
|
Valid Context: |
|
Explanation¶
Tokenisation is a prerequisite for most forms of linguistic annotation. The <w>
element is FoLiA’s basic token or
word (hence the element’s name). This element occurs in the scope of wider structural elements such as <s>
(Sentence Annotation)
Note
This element carries an extra and optional space
attribute with value yes
(default), or no
, indicating
whether a space follows between this token and the next one. This attribute is used to reconstruct the untokenised
text.
Example¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | <?xml version="1.0" encoding="utf-8"?>
<FoLiA xmlns="http://ilk.uvt.nl/folia" version="2.0" xml:id="example">
<metadata>
<annotations>
<token-annotation set="https://raw.githubusercontent.com/LanguageMachines/uctodata/master/setdefinitions/tokconfig-eng.foliaset.ttl">
<annotator processor="p1" />
</token-annotation>
<text-annotation>
<annotator processor="p1" />
</text-annotation>
<sentence-annotation>
<annotator processor="p1" />
</sentence-annotation>
<paragraph-annotation>
<annotator processor="p1" />
</paragraph-annotation>
</annotations>
<provenance>
<processor xml:id="p1" name="proycon" type="manual" />
</provenance>
</metadata>
<text xml:id="example.text">
<p xml:id="example.p.1">
<s xml:id="example.p.1.s.1">
<w xml:id="example.p.1.s.1.w.1" class="WORD">
<t>Hello</t>
</w>
<w xml:id="example.p.1.s.1.w.2" class="WORD" space="no">
<t>World</t>
</w>
<w xml:id="example.p.1.s.1.w.3" class="PUNCTUATION">
<t>!</t>
</w>
</s>
<s xml:id="example.p.1.s.2">
<w xml:id="example.p.1.s.2.w.1" class="WORD">
<t>This</t>
</w>
<w xml:id="example.p.1.s.2.w.2" class="WORD">
<t>is</t>
</w>
<w xml:id="example.p.1.s.2.w.3" class="WORD">
<t>an</t>
</w>
<w xml:id="example.p.1.s.2.w.4" class="WORD" space="no">
<t>example</t>
</w>
<w xml:id="example.p.1.s.2.w.5" class="PUNCTUATION">
<t>.</t>
</w>
</s>
</p>
</text>
</FoLiA>
|