Reviewer has chosen to be Anonymous

Overall Impression: Weak

Content:
Technical Quality of the paper: Good
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Needs improvement
Level of English: Unsatisfactory
Overall presentation: Average

Detailed Comments:

The authors present an enlarged version of their conference contribution, where they test LLMs as knowledge extractors, checking the ability of several LLMs to build elements of knowledge graphs. This is an interesting topic, in some ways not entirely mixing neural and symbolic methods (they happen in distinct moments, the neural part responsible for building symbolic elements that are to be latter used in some appropriate way). The authors present several metrics, some of which are new contributions, and describe interesting experiments. However, some of that seems a bit ad hoc, as discussed in the following paragraphs.

However, the main problem here seems to be the presentation. In fact, the writing is a bit difficult, and several sentences and paragraphs are not entirely clear. I recommend a major revision, where the whole text is rewritten carefully for maximal clarity. I believe the new version that will emerge from that will be a much better manuscript with much higher impact.

First, on questions concerning content and overall presentation.
The main issue concerning presentation is that, in my view, there is a confusing mix of discussion concerning the specific systems developed by the authors and the broad techniques under analysis. It would be important to separate what are the textual parts that discuss the tested ontologies/extractors, and the parts that are really related to LLMs in general. It is actually hard to explain exactly what to do here, but in several paragraphs I was confused as to what the authors were saying: were they describing some specific system that was only used in the specific experiments, or were they discussing the key concepts that are discussed in the experiments? This is the main issue; I suggest the authors read again their text thinking as a new reader, and try to improve the text as much as possible in this regard.
Other points:
- The ontology in Figure 1, used throughout the text, is quite simple; how would conclusions change with a more involved ontology? Any comments?
- The "flexible" metric is an interesting contribution, and the explanations related to it are interesting, but all of it seems to be quite ad hoc and hard to justify. I suggest more discussion is provided.

Now, on the text, some smaller comments:
- The discussion of CRUD operations at the beginning of Page 2 is quite confusing; that paragraph should be rewritten.
- Page 2, line 10: I believe it should "the literature", as an example of a small suggestion that could be applied to several other sentences.
- Page 2, line 12: "is that ... to", seems to be incorrect (remove "that").
- Page 2, paragraph "Therefore, in our..." is too long, it could be divided in two paragraphs at last.
- End of Page 2: several issues are discussed, but it is hard to know what is the actual point of the discussion. What is intended? What are the exact points of the datasets? And so on.
- Page 3, line 4: items 4 and 5 are not part of a pipeline, they seem to be separate modules.
- Page 3, middle of page: the authors write "they", "their", etc, and it is often difficult to know who are the referred entities. This happens a number of times in the paper.
- Definition 2 mixes LLMs and the specific transformer architecture, this is confusing (there are language models that are not transformers!).
- Mathematical expressions should end with period/comma, as appropriate; after a mathematical expression, no indentation is there is no new paragraph.
- What is the meaning of Expression (3)? It just offers an equality.
- Table 1: "manager", not "maager".
- Figure 2 is very hard to understand.
- Page 6, sentence "For a more comprehensive...", is very hard to parse. What does it mean?
- Page 8, "let's ask a model" seems weird.
- Page 9, line 3, "Experts [3]" misses a space.
- Table 7 and Table 8 do not have underline cells; why is it?
- Page 12, "Complex Class Types Do Not"... should be "Does".
- Page 13 mentions "Adhere" but does not seem to agree with the content of the paragraph.

Reviewer has chosen not to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Average

Detailed Comments:

The paper is well-written and explores an interesting topic, focusing on the use of LLM for tasks related to Knowledge Graphs. That said, there are a few areas where the work could be improved to make a stronger contribution:

- Concepts like zero-shot, few-shot, and TOD systems are introduced but not explained in detail. Adding a bit more background would make the paper more accessible to readers who are less familiar with these topics.

- The flexible evaluation approach is innovative, but the rationale for choosing specific penalty percentages isn’t clear. Explaining why certain values were picked would make the methodology more transparent and credible.

- Considering GPT-4’s widespread availability, relatively low cost, and improved performance compared to GPT-3.5, it would be great to see it included in the experiments, especially to test its capabilities in more challenging contexts.

- The experiments conducted feel a bit limited, which reduces the practical contribution of the work. It would be helpful to expand the ontologies to include more complex and realistic scenarios and test on a larger and more diverse dataset to better demonstrate the effectiveness of the flexible evaluation approach.