By Anonymous User
Review Details
Reviewer has chosen to be Anonymous
Overall Impression: Average
Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes
Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Average
Detailed Comments:
The article has been significantly improved from the previous version and the narrative is more cohesive and well articulated. The contribution is now more narrow and clear, and I found the removal of some previous subsection to have improved the scope of the article. Nonetheless, some sections (especially sections 6 and 7) still necessitate more improvement, as described below.
Overall, the approach is flexible and scalable, and the formulation of the new patterns through the reuse and the contextualisation of the previous Boxology patterns as compositional modules further demonstrate the expressiveness of the framework.
The literature mentioned in Section 5 when presenting the use cases also demonstrates a good level of coverage of the proposed patterns.
Pointers for further improvement
--------------------------------
- P2, Line 23. Fine-tuning can also lead to catastrophic forgetting.
- P2. The authors present two common approaches to address hallucinations: fine-tuning and RAG, mentioning that they introduce novel challenges by themselves. However, only the challenges of finetuning are discussed in the paragraph.
- P2. Overall, I got the impression that the limitations of LLMs and Generative models in general are used as motivations of this work, but I do not think this is necessary given the scope of this contribution. I would recommend to focus on providing a short overview of the different methodologies: statistical modelling / fully data driven approaches, symbolic, neuro-symbolic (which all come with their own merits and in their respective fields) to contextualise this work. This short introduction would also be useful to introduce NeSy AI in order to reach a potentially larger audience.
- P2. While I really appreciate the potential of this work, I believe that the introduction still misses a paragraph that comprehensively highlights the motivations of this article and the benefits/potential brought by this work. Currently. the introduction goes straight from the limitations of LLMs to the difficulty of keeping track of an evolving landscape of new models (for which a reader may get different ideas on how to address this problem) and the extension of Boxology. I understand this work extends previous work (which is indeed referenced), but I would appreciate a more detailed presentation of the motivations and benefits of this approach in particular.
- P5. The authors mention (in 3.1 and 3.2) that the proposed pattern can generalise to other modalities, but little information is given on how this would be possible. Providing more insights on how this can be achieved would strengthen these arguments. Also, what about models handling multiple modalities as input?
- Section 4, opening (P7). Re-emphasising again the limitations of LLMs in this section, after this was done in the previous parts, sounds a bit repetitive and may also set the reader off; especially with the strong arguments on LLMs' inability to understand concept of truth, casuality, etc. I believe this section should really focus on the patterns.
- Section 4. While relevant and well-linked with respect to the literature, I still found the description of the patterns a bit disconnected from the diagrams. Adding some basic notational elements from Boxology such as `model:LLM`, `infer:deduce` in \texttt{} while the patterns are presented and explained would significantly improve this connection with little effort.
- Section 5. The current structure of this section focuses on introducing one or more relevant methods for each category/subsection (such as Retrieval Augmented Generation - RAG) and then pointing to the corresponding pattern. However, I found the level of detail of this elaboration a bit non-uniform across the various subsection. For example, the way the pattern covers RAG in 5.1 is clear, whereas other subsections conclude with pointers to figures or other subsections without much detail on how these are actually captured/mapped.
- Section 6, Discussion. While the contributions of the paper are clear, I found that: (i) some arguments remain a bit vague (such as the remark on data and symbols) and would need more elaboration and specificity; and (ii) an overview of the potential scenarios for reuse of this work is still missing. For example, I can see how the Boxology extensions could be reused for illustrating and explaining the models and the processes to a broader audience, and even lead to the identification of their "common inner workings" when it comes to training, inference, generation; or even the possibility of "model checking" ML models? I am trying to be a bit speculative here, just to give an idea of what I was expecting at this stage. Overall, the current discussion section reads a bit rushed, and a more elaborate analysis of this approach, along with the limitations, but also covering what this work actually enables now that we have these new patterns would make the whole contribution stronger.
- Section 7, Conclusions and FW. I strongly encourage the authors to revisit and improve this part, especially the first paragraph. The current version provides very little context on the research problem and the contribution made by this work.
- There are still some typos and incomplete sentences in the document. This is a non-exhaustive list:
- P5, Line 7. "The elementary patterns 2a-2d are patterns that describe to use a model" [incomplete sentence?]
- P9, Line 7, "utelizing" [typo]
- P11, Line 51 "LL-augmented KG" [typo]
