Virtual EZ Grid - Meetings

 

Seminar: 02nd July 2010, 10:00 — 12:00

  • Location: Geneva university.

  • Presents:

    • HES-SO: Marko Niinimaki
    • UniGE: Bastien Chopard, Mohamed Ben Belgacem

  • Invited people:

      Michel C. Milinkovitch, Natan jenkins, Henning Mueller, Antoine Geissbueller, Alain Jacot-Descombes, Gilles Marty, JF Rossignol, Nicolas Ray, Stephane Goyette, Markus Mueller, frederique lisacek, Paola Merlo,Matthias Currat, Jose Nunes, P-Y Burgi, Christoph Berthod, Nicolas Mayencourt, Andreas Malaspinas D Pfenniger, Daniel Agulleiro, PY Morgantini, Alexandros Kalousis, Stephane Marchand-Maillet, Jean-Luc Falcone, nabil abdennadher, Fokko Beekhof

  • Objectives of the seminar presentation:

    Virtual EZ-Grid is a new distributed computing infrastructure funded by AAA/SWITCH. Developed mainly by hepia, the CUI, and DINF, the infrastructure has now reached a sufficient maturity to accommodate projects testing. Similar to a "volunteer computing" platform or "Desktop Grid", Virtual EZ-Grid allows researchers to exploit the power of hundreds of PCs installed at various sites, including hepia, the university of Geneva, university of Neuchâtel and university of Lugano.

get Slides in French [.pptx, .pdf ]

Phone Meeting: 23th April 2010, 11:00 — 12:20

  • Presents:

    • HES-SO: Nabil Abdennadher, Marko Niinimaki, Cédric Bilat, Claud Evéquoz
    • UniGE: Mohamed Ben Belgacem

  • Discussion:

    1. The NeuroWeb application is ported on the platform XWCH (about 45 persistent workers).
    2. The persistent server issues are resolved by HE-ARC and should be validated by hepia.
    3. The memory size estimation and ressource requirements features are integrated on NeuroWeb modules.
    4. What remains to be done:
      • Implementation of the norme L1.
      • Integration of L1 in NeuroWeb modules.
      • Developement of windows solution for JVMs communication.
      • Separation of the Master and client modules.

      Due to deadlines constraints, the decision to postpone the end date of the project should be made by all partners.

see Minutes [in French]

Phone Meeting: 16th April 2010, 11:00 — 12:15

  • Presents:

    • HES-SO: Nabil Abdennadher, Marko Niinimaki, Cédric Bilat, Claud Evéquoz
    • UniGE: Mohamed Ben Belgacem

  • Discussion:

    1. The T12 test algorithm is validated by hepia.
    2. Persistent servers issues have to be resolved. Several different measures have been taken to isolate them.

see Minutes [in French]

March, 22 2010, 13:00 - 17:00

  • Location: Geneva university.

  • Presents:

    • UniNE: Peter Kropf, Lucas Charles
    • UniGE: Bastien Chopard, Mohamed Ben Belgacem
    • HES-SO: Francesco Moresino, Nabil Abdennadher

  • Agenda:

    1. Infrastructure status
    2. EZ Grid module
    3. NeuroWeb gridification
    4. Economic model
    5. Virtual EZ Grid Extension

  • Presentations and discussions:

    1. Infrastructure status: UniGE reported that a VM can be stopped and resumed according to the load of the student computer. "Firedeamon" software is used to control the VM status and the workload of the machines. For "light" applications, Firedeamon keeps the VM running. When the student machine becomes loaded, the VM is suspended.

      We have more than 1000 workers installed. However, due the fact that UniGe machines are not available at night and the VM installation constraints, it's difficult to reach 1000 simultaneously connected workers.

    2. The EZ Grid module: UniNE presented the state of progress of the module implementation. This work will not be integrated at UniGE, due to security constraints reported by UniGe. The proposed solution consists on replacing this module by the Firedeamon software which is already installed in all UniGe machines.

      A study may be carried out to explore the features of this software.

    3. NeuroWeb gridification: a prototype is already deployed on the platform. The next step consists on optimizing this prototype.

    4. Economic model: Francesco Moresino (HES-SO) presented the economic model. Three aspects for the pricing system are discussed:

      • Workers heterogeneity
      • Grid capacity
      • Job definition

      These aspects will be integrated soon.

    5. Virtual EZ Grid Extension: If confirmed, the project will be application driven. A workshop will probably be organized for potentially interested users in order to present the platform and discuss how their scientific applications could be deployed.

    •  

    December, 14 2009

    • Location: Neuchatel university.

    • Presents:

      • UniNE: Peter Kropf, Lucas Charles
      • USI: Cesare Pautasso, Francesco Lelli
      • UniGE: Mohamed Ben Belgacem
      • HES-SO: Markopekka Niinimaeki , Cedric Billat, Francesco Moresino, Nabil Abdennadher

    • Agenda:

      1. Overview of the project, HES-SO, hepia.
      2. The negotiation model and its implementation, HES-SO, HEG. [slides]
      3. Status of the infrastructure platform, UniGe.
      4. The new version of EZ-Grid and its integration with XtremWeb-CH, UniNe.
      5. Security access to the platform, HES-SO, hepia. [slide]
      6. Integration of JOpera and XtremWe-CH, USI. [slides]
      7. Bridging ARC with XtremWeb-CH, HES-SO, hepia. [slides]
      8. The NeuroWeb application ⇒ HES-SO, EI-ARC [slides]
        9. .

    • Presentations and discussions:

      1. Overview of the project (by Nabil Abdennadher).

      2. Negotiation model (by Francesco Moresino): 3 models were presented. Models 2 and 3 are not adapted to the Virtual EZ Grid platform. Currently, the first model is being improved and will be implemented.

      3. Status of the infrastructure: Only 400 are now connected. PCs from UniGE will be available soon (January 2010) only during night. They will be progressively availables 24/24 starting from February 2010.

      4. The development of EZ Grid module and its integration in the XWCH worker: will be achieved during Q1 of 2010.

      5. Security presentation (by Markopekka Niinimaeki): XWCH users are authenticated by Shibboleth technology. This functionality is now completely operational.

      6. JOpera/XWCH presenatation (by Francesco Lelli): This work is 100% achieved.

      7. ARC/XWCH bridge presentation (by Markopekka Niinimaeki): ARC jobs can now be submitted to XWCH.

      8. Presentation of the NeuroWeb application (by Cedric Billat):

        • Discussion about how to gridify the application?
        • How to decribe the workflow in order to use JOpera?
        • Performance in term of CPU and communication: How to split data? When migrate data? Why not using a cluster (with MPI) instead of a Volunteer computing platform (such as XWCH)?: very expensive : All the application has to be written.

      9. End of the project: Planned for September 2010.

    • To do list:

      • Integration of EZ Grid: Planned to be available during Q1, 2010.

      • Connexion of 1000 machines from UNiGE: Will be done progressively during the period Q1 and Q2 of 2010.

      • Gridification of NeuroWeb application: Several versions are planned. The final version will be deployed during Q2 of 2010.

      • Gridification of MedGift: Will strat on January 2010.

      • Implementation of the negotiation model: Planned to start in Q1, 2010.

     

    November, 10 2009.

    • Location: hepia.

    • Presents:

      • hepia: Mohamed Ben Belgacem, Nabil Abdennadher
      • UniNE: Lucas Raphaël Charles, Peter Kropf

    • Agenda:

      1. discussion regarding the hypervisor (virtual machine tool) to use for the worker installation.
      2. features to provide by the external program (EM) in order to monitor remotely the virtual machine.

    • Discussion:

      The hypervisor to use is the "Virtual Box" since UniGE used it on students' machines. In this context, UniNE tasks is to develop the EM to monitor remotely the virtual machine where the worker is installed. The to do list is as follow:

      1. Design architecture:

        • Virtual machine (VM) is installed on each machine.
        • On the same machine, an external module (EM) is installed to monitor the VM: Shutdown, restart, pause, resume, checkpoint.
        • External XWCH-server (ES) sends commands to the EM. the EM sends periodically a "heartbeat" back to the ES.

      2. For XWCH-EZ Grid integration, 3 steps are fixed:

        • Study the API provided by SUN Virual Box hypervisor.
        • Implement and test these features: shutdown, restart, pause, resume, local checkpoint.
        • Implement the ES remote commands.
        • Implement the remote checkpointing.
        • Define the communication protocol between the XWCH-scheduler and the ES.

      3. Extra-to do:

        • Inturreption system detection when saving status for a VM.
        • Data integrity.
        • performance comparison between VirtualBox and VMPlayer.

     

    Phone Meeting: 15th October 2009, 9:30 — 10:30

    • Presents:

      • UniNE: Peter Kropf
      • USI: Cesare Paustasso
      • hepia: Nabil Abdennadher

    • Discussion:

      1. To Do list for this quarter (October-November-December):

        1. Integration of EZ-Grid and XtremWeb-CH for UniNE. A phone meeting will be setup soon to discuss this point.
        2. Publication of the next release of JOpera (This new version of JOpera supports XtremWeb-CH) + produce the necessary documentation for USI.
        3. First implementation of the economical negotiation model for (HES-SO, HEG).
        4. Development of the C++ API of XtremWeb-Ch for (HES-SO, heig-vd).
        5. Start of the deployment of NeuroWeb application for HES-SO, EI-ARC + hepia).
        6. Deploy XtremWeb-CH on 500 machines (at least) for mainly UniGE : to be confirmed by Bastien, see point 2.

      2. Status of the current machines connected to the platform:

        1. The number of connected workers is around 300 (150 from hepia, 100 form UniGe, 50 form UniNe and heig-vd). We should reach at least 500 machines by the end of the year. So far, we are far from the 1500 nodes we have announced in our proposal.
        2. UniNE will provide 20 nodes.
        3. USI will provide 20 nodes.

      3. Next face-to-face meeting: Monday 14th December (13:00 to 17:00) in Neuchatel. Proposed Agenda : discuss the status of the project + presentations of the developments carried out in the framework of the project.

    • End of the phone meeting

      • Nabil concluded:

      "I have created a new mailing list for the project: virtual-ez-grid@swing-grid.ch . It’s based now on the Sympa server of SwiNG. This will allow us to dynamically manage this list without any dependencies with any IT support.

      Please take note that the web site of the project is: http://www.xtremwebch.net/Projects/Virtual_EZ_Grid/EZ_Home.html. The banner will be changed soon. If you have any ideas regarding the graphic design, please let me know.

      Christoph Witzig would like to "visit" the project. If you agree, I’d like to propose him attending the next face-to-face meeting. What do you think?

      To Bastien: Cesare and Peter already gave their OK for point 3. Could you please tell me if you are fine with the date of Monday 14th December? Your presence is very important since we will discuss the WP related to the infrastructure.

      To all: Please let me know your feedback before Wednesday 4th November, 17:00. I plan to send these minutes to the mailing list and announce the date of our next f2f meeting on Thursday (5th November) morning."

     

    June, 3 2009

    • Location: hepia.

    • Presents:

      • hepia: Markopekka Niinimaeki, Nabil Abdennadher
      • UniGE: Jean-Marc Naef, Jean FranÇois Rossignol, Gilles Marty

    • Agenda:

      1. discussion regarding the infrastructure that will be available ⇒ 900 machines from UniGE.
      2. deployment of 200 machines from UniGE before June 2009, only during night, very basic deployment (no check pointing aupport, etc.).

     

    March, 13 2009

    • Location: hepia.

    • Presents:

      • hepia: Nabil Abdennadher
      • UniGE: Alexandre Le Bouthillier, Jean-Marc Naef

    • Agenda:

      1. implemtation of the EZ Grid tool, discussion regarding the VM tool to be used.

     

    kick off meeting: February, 06 2009

    • Location: hepia-Geneva

    • Presents:

      • UniNE: Peter Kropf
      • UniGE: Jean Marc Naef, Xin Zhou
      • USI: Cesare Pautasso, Francesco Lelli
      • HES-SO: Claude Evequoz, Regis Boesch, FranÇois Tamone, Nabil Abdennadher

    • Agenda:

      1. Introduction: the project and the partners (by Nabil).

      2. Infrastructure & EZ Grid tool (By Jean-Marc Naef): Jean-Marc Naef presented the tool EZ Grid developed by UniGe. EZ Grid I enables users to submit their jobs to the Grid platform and deploy automatically the virtual machines on the computers. For further details see slides in attachment. [slides]

      3. ARC (By Xin Zhou): ARC is a meta-middleware. Nodes are assumed to be clusters of computers and not “simple” computers. For further details see slides in attachmentARC (By Xin Zhou). ARC is a meta-middleware. Nodes are assumed to be clusters of computers and not “simple” computers. For further details see slides in attachment. [slides]

      4. XtremWeb-CH (By Nabil): XWCH is Volunteer Computing middleware. Nabil presented the new version of XWCH which based on a Web services architecture. For further details see slides in attachment. [slides]

      5. JOpera (By Cesare Pautasso): Cesare presented the JOpera workflow system. JOpera enables users describing their application by a data flow graph. For further details see slides in attachment. [slides]

      6. Economic model (By Francesco Moresino).

    • Discussions:

      ARC is a meta-middleware and not a “simple” middleware. An ARC node is not a “computer” or a “workstation” but a cluster of computers supporting a middleware. In this context ARC should be placed on the top of XtremWeb and not at the same level.

      Tasks dealing with tools integration should be reviewed according to the state of the: JOpera, EZ-Grid and XtremWeb.

    • Next meetings:

      Doodle poll will be set up for a phone meeting (planned during March). The next in person meeting could be planned during the first week of May (GPC’09 conference).