Reviewer has chosen to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Good

Detailed Comments:

Undoubtedly, this is a very interesting paper. Essentially, along the lines of Dreamcoder, the paper proposes something calledcoder. The idea is to use program synthesis for abstract task generation and solving Like Dreamcoder and a host of other papers, they introduce a domain-specific and discuss how it can provide the constructs to train a program to solve these reasoning tasks. They tackle the ARC reasoning tasks, which are challenging and worthy of consideration.

Overall, in terms of the project's goals and ambition, I have issues at all. What I do somewhat struggle with is that, unlike those other papers, this article is specifically submitted to the Neurosymbolic Journal. In this case, I would expect a bit more emphasis on the formal machinery behind all of these constructions. The way the paper is introduced is somewhat high-level, presenting some machine learning constructs and terminology without much detail. In fact, the paper is largely textual. Given the venue, I would expect a greater emphasis on semantics, syntax, and the assumptions behind some of the modeling of the distributions and the correctness of the whole thing.

I understand that as a proof of concept, the pipeline is coming together nicely, but as a formal object, there's much to be desired in the writing. I recommend that the authors stay with this piece of work but try to provide some kind of soundness, completeness, or some sort of formal structure to how the various pieces fit together. I understand that this is challenging, but that's what would make it worthwhile for this journal.

Reviewer has chosen not to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Average

Detailed Comments:

The paper presents a neurosymbolic system designed to tackle and solve tasks from the ARC dataset. The system architecture is modular, combining symbolic and subsymbolic components, including a DSL, to generate synthetic tasks and iteratively train a model to solve a subset of these generated tasks alongside the original tasks.

The paper presentation and organization are good. The language is correct with only minor revisions required. The work also explains why the research should be placed in the scope of neurosymbolic systems.

Some points to raise that require the attention of the authors and might need addressing:
- Section 3.2 describes the solver's distributional generation of programs as solutions to given tasks and mentions capturing the relevant semantics needed in addition to the syntax for solving tasks. I am unsure to what extent language-building with grammars can be circumvented and how robust probabilistic semantics might be in replacing them. It seems like there remains a missing link here between syntax and semantics that is not thoroughly covered by the methodology employed.
- Section 3.4 details the DSL created for the purpose of the experiments and the tasks, and the language seems like an interesting bridge between lower-order types and higher-order functions. But since the DSL must deal with different kinds of operations (e.g., general arithmetic versus domain-specific filtering or rotation), it is difficult to see the motivation behind creating a DSL instead of using existing methods like Inductive Logic Programming or existing languages like Prolog to do the job. The motivation for implementing the language from a practical (i.e., this is a language that will have to scale if applied to other problems) and effective perspective (i.e., newly created and for the purpose of this task versus more robust pre-existing languages that are proven to handle logical problem framing and solving) does not seem sufficiently strong and persuasive.
- Section 3.6 presents the training of TransCoder. It is explained that the RL method is not useful at the start of training and kicks in later. It would have been interesting to have a comparison between using RL versus using sole SL training which is more straightforward to implement and effective from the start. It would also help to convince of the criticality of having the RL method on top of the SL method.
- Section 4 mentions related works and addresses LLM technologies, stating that their advancement has enabled better performances at solving ARC tasks than any DSL. This claim raises questions: why haven't LLMs been included in the experiments and compared as benchmarks to TransCoder? Why wasn't LLM technology considered as part of the design of the TransCoder modules? They would have possibly made for a less tedious implementation and more performant solution than a DSL according to the stated literature.
- In Section 5, there is mention of a normalization step without real justification of the importance of this step beyond grouping code into other code logic. The authors mention reducing lines of code but in the examples given it seems more like code substitution and a tedious and unnecessary process to include.
- Figure 5 is hard to read and may be more exploitable if separated into 2 figures (one for SolveRate and one for SynthRate).
- Table 5 is hard to read and I am not sure how to understand it. The explanation seems to suggest there is gradual learning (including of historical tasks), but the table presentation makes it very hard to render these kinds of conclusions.