Submitted projects - expanded view

  • CERN-Solid and the Slides' app

    Project description

    In the CERN IT/CDA group a lot of work was done in 2020 to understand the Solid project ecosystem. Solid is here to stay and develop tools that give the users sovereignty over their own data. This is why we need to continue this work and extend the Proof of Concept (PoC) to other CERN Open Source applications, involving Authorisation/Authentication and user profiles. Most importantly, we need to bring up our own Solid server to host our pods, to be sure that all Data Protection issues are addressed and we have full control over the storage of our data on the pods. See HERE a policy document that explains this need.

    During the same year, we developed a web-based slides' maker, that could become the cross-platform open solution for the community. Some work is still needed to make this application production quality and acceptable by the users. Additional tips or web-based slide makers can be found in the references, including open source projects related to Solid, which can lead to a nice combination of these two efforts.

    This project is about pursuing these two objectives in parallel. Namely:

    For the CERN-Solid collaboration future:

    Based on the conclusions from the successful Proof of Concept (slides, video, transcript, thesis HERE) and the Policy document:

    1. Evaluate/complete the results of MSc project https://it-student-projects.web.cern.ch/projects/cern-solid-server-hosting
    2. Evaluate/document the pod inspecting tool Penny, as a more functional way to access/update one's pod. Make usability recommendations on Penny for future use. 
    3. Design an attractive pod User Interface (UI), compliant with the Solid specifications and reserved fields. Discuss with the CERN-Solid community, in the relevant gitter channels, the relevance of developing a UI in-house.
    4. Create pods for all CERN users and the necessary documentation and/or videos to promote their adoption by the CERN community.

    For the CERN web-based Slides' app:

    1. Process the pending improvements registered in JIRA.
    2. Encourage users to use the app and collect feedback.
    3. Enhance functionality based on other such tools (see References).
    Duration
    1 year
    Contact
    Maria Dimou
    Status
    Submitted
  • CERN Academic Training web site

    Project description

    The CERN Academic Training lectures contain brilliant material of leading-edge technology as well as great historical value. The lectures are open to all members of CERN personnel (in particular staff members and fellows, associates, students, users, project associates and apprentices) free of charge.

    Each lecture is recorded and published on the web along with the visual support material. The complete catalogue of Academic Training and Summer Student Programme lectures are archived since 1999.

    Similar CERN lectures are available via other CERN lecture categories, e.g. Colloquia, Seminars, various students' programmes.

    The project is about designing and building an attractive website to promote CERN's Academic Training, seminars, colloquia and summer student lectures in a coherent way, and easily maintained in future.

    Information should be exchanged with IR/ECO to avoid effort duplication and encourage the site exploration by various target groups. Steps:

    1. List and evaluate what other research and academic institutes do to promote their educational material. An inventory was done in 2016 here that needs to be enhanced. Check the Royal Institute site, EPFL, MIT and others to select best features. Check the twiki link in the References for input.
    2. Make a mockup of a web site to propose to the Academic Training Committee (ATC). Propose various scenarii - e.g. inclusion (or not) of other CERN lecture categories (by early October).
    3. Make a proposal on the technology to use (it must be an Open Source solution).  For example: Explain why Drupal instead of Wordpress or React. (mid-October). Make sure it works also on mobile devices.
    4. Build the web site https://cern.ch/academictraining with tailor-made views. For example: The lectures in Indico and their videos in the CERN Document Server on "the Higgs boson", "Dark matter", "Machine Learning", "Statistics" etc.
    5. Evaluate and discuss with the Academic Training Committee the possibility to email periodic notifications on the programme activities.
    6. Make a proposal on whether, where and how to manage "Comments". Use the experience obtained from the CERN-Solid collaboration for this. See here the relevant project.
    7. Foresee a button for submission of ideas for future lectures.
    8. Make a proposal for  support on user queries related to the web site.
    9. Foresee an Administrator's documentation for the site maintainer, written in Markdown as per this guide.

    Note: The following points were included in the original project proposal. They will be addressed via other projects with other funds.

    1. Using the same Markdown-based documentation policy make a technical recommendations' document for the end-users https://makeyourownsite.docs.cern.ch, with instructions, based on the technologies available from IT/CDA-WF.
    2. Simplify the IT/CDA-IC and IT/CDA-DR workflow at lecture video publishing time, in collaboration with the weblecture-service and CDS and videos.cern.ch experts.
    3. Help with  the migration of the Academic Training  videos' collection from CDS to videos.cern.ch.
    4. Help with the bulk update of this videos' backlog with subtitles, based on the output of the related effort to select a tool for this purpose.
    Duration
    one year
    Contact
    Maria Dimou
    Status
    Submitted
  • Deploy subtitles as a service for CERN videos

    Project description

    2021 version of the proposal

    After the Tender selection process completes for a live and offline transcription and translation product,
    help to post-process and automate the  enhancement of the lectures' backlog with subtitles.
    See here categories of candidate lectures.

     

     

    2020 version of the proposal (TECH approved by IT and HR, rejected by the service manager):

    The CERN IT Collaboration, Devices & Applications group and in particular sections Digital Repositories (IT CDA/DR) and Integrated Collaboration (IT CDA/IC), run many highly visible and popular services, which enable researchers/institutions to share and preserve their research data, software and publications as well as meet, present and record lectures, projects, plans and decisions of academic content and very large experiment collaborations.

    The CERN Document Server (CDS) is the official document repository for the laboratory and annually serves around 2 million visitors. There are thousands of videos recorded at CERN via the CDA/IC recording and transcoding infrastructure. They are uploaded and viewable via CDS or the recent videos' portal.

    We need to equip all CERN-made videos with subtitles. This project is about turning the transcription software, to be selected by the relevant CERN CDA service managers, into a scalable service that automatically introduces and displays subtitles in CDS for the CERN community. The process should be well integrated with our new video player and the set-up should allow to apply text corrections by the content owner (lecture, meeting, conference organiser) in a functional way.

    Duration
    12-14 months
    Contact
    Maria Dimou
    Status
    Submitted
  • Documentation Project - Migration of static web sites to Markdown

    Project description

    The group Collaboration, Devices and Applications (CDA) in CERN IT Department initiated a Documentation Project for moving static service documentation web sites containing service documentation to Markdown. For this purpose, software tools are being developed, instructions being written and  pilot testers contacted.

    The student will have to:

    1. apply the instructions in order to create a new web site according to the policy for site naming, hosting, authoring.
    2. observe new-comer users' reactions to the procedure, the tools and the suggested rich-text editors.
    3. identify shortcomings in any area of point 2 above and report to the documentation-project members.
    4. document tips in form of FAQs and/or discourse articles.
    5. make recommendations to the documentation-project team for an optimal user experience.
    6. disseminate the instructions via alternative methods, namely online tutorials.
    7. convert some of the groups' own service documentations to test the process in operation.
    8. make short videos based on the Instructions.
    Duration
    6 months
    Contact
    Maria Dimou
    Status
    Submitted
  • Analysis of the I/O performance of LHC computing jobs at the CERN computing centre

    Project description

    The LHC experiments execute a significant fraction of their data reconstruction, simulation and analysis on the CERN computing batch resources. One of the most important features of these data processing jobs is their I/O pattern in accessing the local storage system, EOS, which is based on the xrootd protocol. In fact, the way experiment applications access the data can have a considerable impact on how efficiently the computing, storage and network resources are used, and has important implications on the optimisation and size of these resources.

    A promising approach is to study the logs of the storage system to identify and characterise the job I/O, which is strongly dependent on
    the type of jobs (simulation, digitisation, reconstruction, etc.). A direct link between the information in the storage logs and the information in the monitoring systems of the experiments (which contain detailed information about the jobs) is possible, as it can be derived from a cross analysis of the aforementioned data sources together with information from the CERN batch systems. The goal of this project is to study such connection, use it to relate I/O storage patterns to experiment job types, look for significant variations within a given job type, identify important sources of inefficiency and describe a simple model for the computer centre (batch nodes, network, disk servers) that would increase the efficiency of the resource utilisation.

    In case inefficiencies are detected that could be alleviated by changes in the way experiments run their jobs, this information should be passed to the experiments.

    The analysis can be initially based on the jobs of a single large LHC experiment (ATLAS or CMS) and extended to other experiments if time allows.

    Duration
    3 to 6 months
    Contact
    Andrea.Sciaba@cern.ch
    Status
    Submitted
  • Optimisation of experiment workflows in the Worldwide LHC Computing Grid

    Project description

    The LHC experiments perform the vast majority of the data processing and analysis on the Worldwide LHC Computing Grid (WLCG), which provides a globally distributed infrastructure with more than 500k cores to analyse the tens of PB of data collected each year. Profiling of the computing infrastructure with respect to the impact of different workloads is a crucial factor to find  the most efficient match between resources and use cases.  From the current analysis it is clear that the efficiency is neither perfect nor well understood.

    There is a rich amount of information collected by the communities' monitoring infrastructures. The scale and complexity of this data presents an analytics challenge on its own. So far the full potential hasn't been exploited. This data covers all the areas of the computing activities such as host monitoring, storage, network, batch system performance, user level job monitoring. Extracting useful knowledge from this data requires the use of state of the art data analytics tools and processes. The final purpose is to gain deep understanding of what determines the efficiency and how it can be improved. 

    ElasticSearch is a distributed, search and analytics engine  that is used at CERN to store and process large amounts of monitoring data for several experiments.

    It has been noted that differences in data access patterns lead to significantly different utilisation of the resources. However, the concrete causes and quantitative relations are still badly understood. In the same way job failures due a variety of underlying causes lead to loss of productivity, without knowing the exact causes and the concrete scale of the different issues.  

    To be able to improve the overall efficiency we suggest to studying the dependency of the performance on a variety of variables. Based on these findings, which could be obtained by classical and/or machine learning based data analysis techniques, new strategies should be developed. Since the expected gains are on the order of 10-20% the outcome of this work is of great importance for the experiments and the infrastructure providers. 

    The work in this project will be done in close collaboration with experts from CERN IT and the LHC experiments. 

    Duration
    3 to 6 months
    Contact
    Andrea.Sciaba@cern.ch
    Status
    Submitted
  • Advanced Notifications for WAN Incidents

    Project description

    One of the main challenges in WLCG WAN networking is the network diagnostics and advanced notifications on the issues seen in the network. LHCOPN/LHCONE as the core global networks in WLCG have more than 5000 active links between 120 sites. Currently, most of the issues are only visible by the applications and need to be debugged after the incident and performance degradation has already occurred. This is primarily due to the underlying complexity of the WLCG network (multi-domain) and difficulty to understand state of the network and how it changes over time. The project will aim to use the current open-source event processing systems to automate detection and location of the network problems using the existing data from the perfSONAR network infrastructure. The project will be done in collaboration with University of Chicago and University of Michigan.

    The project will build on the standard WLCG perfSONAR network measurement infrastructure and will aim to gather and analyze complex real-world network topologies and their corresponding network metrics to identify possible signatures of the network problems. It will provide a real-time view on the existing diagnosed issues together with a list of existing downtimes from the network providers to the experiments operations teams.

     
    Duration
    12 months
    Contact
    Marian.Babik@cern.ch
    Status
    Submitted
  • Dynamic storage federations

    Project description

    The group runs a project whose goal is the dynamic federation of

    • HTTP based storage systems, allowing a set of globally distributed resources to be integrated and appear via a single entry point. The task is to work on the development of this project (“dynafed”), implementing functional and performance extensions, in particular 
    • Redirection monitoring, to allow the logging of federator behaviour for real-time monitorng and subsequent analytics
    • Metadata integration, beginning with the incorporation of space usage information, allowing the federator to expose grid-wide storage metrics
    • An endpoint status/management subsystem. The basic feature would be an interface that publishes endpoint status (up/down/degraded). Management functions could also be incorporated, including ways to add/enable/disable endpoints without having to restart the service.
    • Semantic enhancements to the embedded rule-based authorization implementation, including turning the authorization subsystem into a pluggable authorization manager.
    • Deployment tests and development with other Apache security plugins, to support natively Identity Federations, like the CERN SSO, Facebook, Google and others. May benefit from the previous points about authorization.
    • Integration with experiment catalogues to benefit from available metadata and replica placement information.
    Duration
    From 3 to 9 months depending on a selected task
    Contact
    oliver.keeble@cern.ch
    Status
    Submitted
  • Distributed storage systems for big data

    Project description

    The group maintains a framework called dmlite which is used to integrate various types of storage with different protocol frontends. It is the basis of a number of the group’s products such as the Disk Pool Manager (DPM), a grid storage system which holds over 50PB of storage in the global infrastructure. DPM/dmlite extensions
    The task is to contribute to the dmlite project by working on functional extensions to the framework. Example projects include

    • Exposing system data through a “procfs” style plugin
    • Incorporation of new AA mechanisms, eg outh
    • Creation of a web admin interface
    • Work on draining and file placement within the system
    • dmliteSE

    Help to realise the group's vision of a “dmliteSE” by working on the gradual retirement of legacy daemons within the DPM system. In this context, tackle the modernisation of pool management and file placement, and the incorporation of new resource types (eg cluster file systems) into the system. Complete the functional development required to allow operation of a disk storage system purely through standard protocols.

    Duration
    3 to 9 months depending on the selected task
    Contact
    oliver.keeble@cern.ch
    Status
    Submitted
  • File Transfer Service (FTS) extensions

    Project description

    The File Transfer Service (FTS) manages the global distribution
    of LHC data, moving multiple petabytes per month during a run and underpinning the whole data lifecycle. Join the FTS team in their development of this critical service. Possible projects include

    • authorised proxy sharing: allowing a production service to delegate a proxy and authorising others to use it
    • incorporation of support for new types of endpoint, for example cloud or archival storage
    Duration
    From 3 months, depending on task selected
    Contact
    oliver.keeble@cern.ch
    Status
    Submitted
  • Performance optimization in a High Throughput Computing environment

    Project description

    Profiling of computing resources respect to WLCG experiment workloads is a crucial factor to select the most effective resources and to be able to optimise their usage.
    There is a rich amount of data collected by the CERN and WLCG monitoring infrastructures just waiting to be turned into useful information. This data covers all the areas of the computing activity such as (real and/or virtual) machine monitoring, storage, network, batch system performance, experiment job monitoring.
    Data gathered by those systems contain great intrinsic value, however information needs to be extracted and understood through a predictive data analytics process. The final purpose of this process is to support decisions and improve the efficiency and the reliability of the related services.
    For instance, with the adoption of the remote access of data it becomes mandatory to understand the impact of this approach to the job efficiency. Here the interplay of network and CPU effects, as well as the resource usage from multi VOs needs to be studied and understood. An interesting topic of study is the performance of job processing at the WLCG distributed T0 center, which is physically split between Computer Centers in Meyrin and Wigner. The goal of the project will be to understand the difference in the performance and to suggest possible optimization.

    The work will be conducted in close contact with the experts (CERN analytics working group, system managers, developers) and will provide a deep insight into the computing infrastructure of a WLCG datacenter, its design, technical requirements and operational challenges.

    Duration
    6 to 12 months
    Contact
    julia.andreeva@cern.ch
    Status
    Submitted
  • QA in distributed cloud architecture: evolution of smashbox framework

    Project description

    Cloud synchronization and sharing is an area in evolution with innovative services being built on top of different platforms. CERNBOX is a service ran at CERN to provide at the same time synchronisation services (based on the OwnCloud software) and high-performance data access and sharing (based on EOS, the CERN disk storage system for large-scale physics data analysis).

    The Smashbox framework (https://github.com/cernbox/smashbox) is successfully used on Linux clients to test OwnCloud/CERNBOX installations. The plans to extend it require to port it to non-Linux platforms:
    * Smashbox port to Windows platforms 
    * Smashbox port to Android 
    * Smashbox port to iOS 
    * Smashbox orchestration (concurrent execution across platforms)

    Duration
    3-12 months depending on the agreed scope
    Contact
    massimo.lamanna@cern.ch
    Status
    Submitted
  • Cloud data analysis

    Project description

    Cloud synchronization and sharing is a promising area for the preparation of powerful transformative data services. 

    The goal of this project is to prepare CERNBOX to be used in connection with heavy-duty activities (large-scale batch processing) on the current LXBATCH infrastructure (LSF) and on its future evolution (HT-Condor): physicists can enable their data to move across their private workstations (like a private laptop) while the bulk of the data is directly accessed from the EOS infrastructure. At the same time, users can control the progress of their activity via mobile clients (as a smartphone) via optimised client applications or via standard browsers.
    The student will participate to the preparation and validation of these use cases. The student will participate to the deployment of the necessary infrastructure (EOS Fuse access from interactive and batch services), support the alpha users (physicists) and extend the current testing and validation system to these new use cases and to new platforms (acceptance tests – in connection with other sites running CERNBOX and monitoring – using the CERN monitoring infrastructure).

    An additional use case is the enabling of data viewers (ROOT tuple) in connection with the SFT team to allow seamless access to summary data (like histograms) from the CERNBOX portal directly.

    Duration
    6-12 months
    Contact
    massimo.lamanna@cern.ch
    Status
    Submitted
  • QA in distributed cloud architecture: injection-fault testing

    Project description

    Clients of the sync&share system (CERNBOX) are particularly exposed to "operational failures" due to heterogeneity of hardware, OS and network environments. 

    Sync&share system operates in very heterogenous network environment: from fast, reliable network inside the computing center to unreliable, high-latency ad-hoc connections such as from air-ports etc. 
    Windows filesystems have substantially different semantics (e.g. locking) from Unix filesystems -- these difference affect the synchronization process 
    the goal of the R&D is to analyze the environment and identify the relevant classes of failures in order to provide a reproducible framework for injecting faults at the system level for testing client-server data transmission 
    examples: 
    * network slowdown or packet loss 
    * local disk failure 
    * checksum errors 
    * failed software upgrades 
    the work is supported by real monitoring and logging data: failure patterns in an existing service (CERNBOX) 
    the work extends on existing testing framework (smashbox) 

    Duration
    6 months
    Contact
    massimo.lamanna@cern.ch
    Status
    Submitted