Token Annotation¶
This annotation type introduces a tokenisation layer for the document. The terms token and word are used interchangeably in FoLiA as FoLiA itself does not commit to a specific tokenisation paradigm. Tokenisation is a prerequisite for the majority of linguistic annotation types offered by FoLiA and it is one of the most fundamental types of Structure Annotation. The words/tokens are typically embedded in other types of structure elements, such as sentences or paragraphs.
Specification¶
Annotation Category: | |
---|---|
Declaration: |
|
Version History: | |
Since the beginning |
|
Element: |
|
API Class: |
|
Required Attributes: | |
Optional Attributes: | |
|
|
Accepted Data: |
|
Valid Context: |
|
Explanation¶
Tokenisation is a prerequisite for most forms of linguistic annotation. The <w>
element is FoLiA’s basic token or
word (hence the element’s name). This element occurs in the scope of wider structural elements such as <s>
(Sentence Annotation)
Note
This element carries an extra and optional space
attribute with value yes
(default), or no
, indicating
whether a space follows between this token and the next one. This attribute is used to reconstruct the untokenised
text.
Example¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | <?xml version="1.0" encoding="utf-8"?> <FoLiA xmlns="http://ilk.uvt.nl/folia" version="2.0" xml:id="example"> <metadata> <annotations> <token-annotation set="https://raw.githubusercontent.com/LanguageMachines/uctodata/master/setdefinitions/tokconfig-eng.foliaset.ttl"> <annotator processor="p1" /> </token-annotation> <text-annotation> <annotator processor="p1" /> </text-annotation> <sentence-annotation> <annotator processor="p1" /> </sentence-annotation> <paragraph-annotation> <annotator processor="p1" /> </paragraph-annotation> </annotations> <provenance> <processor xml:id="p1" name="proycon" type="manual" /> </provenance> </metadata> <text xml:id="example.text"> <p xml:id="example.p.1"> <s xml:id="example.p.1.s.1"> <w xml:id="example.p.1.s.1.w.1" class="WORD"> <t>Hello</t> </w> <w xml:id="example.p.1.s.1.w.2" class="WORD" space="no"> <t>World</t> </w> <w xml:id="example.p.1.s.1.w.3" class="PUNCTUATION"> <t>!</t> </w> </s> <s xml:id="example.p.1.s.2"> <w xml:id="example.p.1.s.2.w.1" class="WORD"> <t>This</t> </w> <w xml:id="example.p.1.s.2.w.2" class="WORD"> <t>is</t> </w> <w xml:id="example.p.1.s.2.w.3" class="WORD"> <t>an</t> </w> <w xml:id="example.p.1.s.2.w.4" class="WORD" space="no"> <t>example</t> </w> <w xml:id="example.p.1.s.2.w.5" class="PUNCTUATION"> <t>.</t> </w> </s> </p> </text> </FoLiA> |