The AIthena consortium partners met in-person in Eindhoven (the Netherlands), and online on 15-16 November. The meeting was kindly hosted by the Eindhoven University of Technology.
For two days meeting, the AIthena partners discussed status of work in each of the Work Packages, as well as upcoming activities and challenges. On the second day of the consortium meeting, the 1st AIthena workshop took place, and consisted of two parts:
- Session 1: User survey results.
- Session 2: Data and Model Cards.
1st AIthena workshop – summary of session 1
The first session of the workshop dealt with the results of the survey that was administered to gather valuable insights into user perspectives, needs, and expectations in the realm of Artificial Intelligence (AI) and Cooperative, Connected, and Automated Mobility (CCAM). The session aimed to leverage insights from external speakers, foster discussions on user needs and challenges, and facilitate breakout sessions for refining the AIthena use cases.
External Speakers: two distinguished external speakers, Matina Loukea from CERTH, Drive2theFuture project, and Dr. Konstantinos Gkiotsalitis from the National Technical University of Athens from CONDUCTOR, shared their expertise and insights with the AIthena project team.
Presentations and discussions: the presentations focused on user needs, challenges, and expectations in the context of AI and CCAM. The discussions following the presentations provided a valuable opportunity for the project team to gain diverse perspectives and insights, enriching the ongoing research on user perspectives and acceptance.
Breakout sessions: following the presentations and discussions, the workshop transitioned into breakout sessions. Three groups were formed, each dedicated to a specific use case: Use Case 1, Use Case 2, and Use Cases 3 and 4 (information about AIthena Use Cases). The goal was to harness the survey results and insights from the external speakers to refine and shape the identified use cases. In these sessions, team members actively engaged in collaborative discussions, exploring how the survey findings could be effectively utilized to enhance the relevance and applicability of each use case. The breakout sessions provided a platform for in-depth analysis and strategizing, fostering a collaborative environment among team members.
Key outcomes:
- Diverse Perspectives: The insights from both external speakers broadened the perspective of the AIthena team, offering diverse viewpoints on user needs and challenges in the context of AI and CCAM.
- Collaborative Refinement: The breakout sessions facilitated collaborative refinement of the use cases, ensuring that the survey results were translated into actionable strategies for the development and implementation of AIthena.
- Enhanced Research Approach: The workshop’s outcomes contribute significantly to the ongoing research, refining the focus areas based on real-world user expectations and challenges.
1st AIthena workshop – summary of session 2
In this workshop, we discussed the key concepts of Data and Model Cards and their use in the AIthena project. Data and Model Cards are reports that provide important information about a dataset or a Machine Learning (ML) model that we have trained. These were first introduced by Google, who suggested that for AI to be more user-focused, we need to inform users, developers, and auditors about the specific information in a dataset, how a model was trained, what results can be expected, and where improvements can be made. These documents contribute to explainability, transparency, governance and, in general, to the trustworthiness of ML models.
Data Cards includes information about a dataset, such as who published it or who funded the process. It also includes details about the data, like what is recorded, the date, and importantly, some data statistics that can help us see if every class is well represented. For example, if the data doesn’t include many cyclists, a model trained with this data might not recognize them well. The Data Card also needs to mention any sensitive data that was recorded, whether intentionally or accidentally, and explain how we handle such data.
Continuing with Model Cards, like a Data Card, they include details about the model’s development, performance, limitations, risks, purpose, among others. The evaluation included in the model car must have metrics of the model against different demographic groups to assess the model’s fairness and to identify any potential biases, if the model purpose affects humans. For example, if a facial recognition model performs well on light-skinned individuals but poorly on dark-skinned individuals, this would be clearly indicated in the Model Card. This allows users, developers, and auditors to understand the model’s limitations and potential areas for improvement.
For AIthena, we have proposed specific versions of Model and Data Cards. These are designed to serve the same purpose as those proposed by Google, but with a focus on the CCAM (Cooperative, Connected, and Automated Mobility) field of application. Hard and digital copies of this documents were distributed to the attendees of the workshop to gather feedback about the usefulness, comprehensiveness, understandability, and the representativeness of the sections included in these documents.
