Tuesday, March 11, 2014

Synnefo v0.15 Released

Hello Synnefo users,

we are happy to announce that Synnefo v0.15 is out!

This release brings major new features and significant bug fixes across Synnefo.
It focuses mostly on Networking and the changes there affect the whole stack, so you will also see new exciting things on your Web UI.

Note that starting v0.15, Ganeti nodes do not need to have access to the Pithos DB anymore, so this means that Ganeti clusters are now completely separated from the upper Synnefo layer, boosting scalability and security. Additionally, it is now really easy to have Ganeti clusters on geographically distinct locations.

More features are also included, copying/pasting from the NEWS file:

Cyclades
  • Networking: Major improvements in the Network Service.  This release introduces a complete implementation of the OpenStack Networking (Neutron) API v2.0, with distinct /networks, /ports, /subnets, and /floatingips API endpoints, replacing the old Cyclades /networks API.
  • Networking: Support the creation of arbitrary L2/L3 network topologies, with IPv4-only, dual IPv4/IPv6 or IPv6-only functionality.
  • Networking: Simplify network administration, support forced networking configurations, e.g., a permanent management network across VMs.
  • Networking: Introduce new CLI management commands for handling networks, subnets, and ports.
  • Networking: Support management of floating IPs as a distinct resource in Cyclades. Users may create new IPs from admin-defined IP pools, attach them to and detach them from VMs arbitrarily, and eventually release them.
  • Compute: Support server resize, by changing server Flavors. Extend the API and UI accordingly. Support fine-grained, per-flavor control of resize and create actions ("allow_create" flavor attribute).
  • Compute: Support finer-grained accounting of "Active CPUs", "Active RAM"  resources, in addition to "CPUs", "RAM". Active resources are those of running VMs. This allows the administrator to limit currently used VM resources to only a subset of the total allocated resources.
  • Compute: Support NIC hotplugging. Allow addition and removal of NICs on running VMs, without a reboot, to support on-the-fly network reconfiguration, e.g., allocation of floating IPs, without the need for NAT.
  • Compute: Speed up server reconciliation, by performing operations in parallel, across Ganeti backends.
  • Compute: Support running snf-vncauthproxy in distinct machine, with control channel over TCP. This simplifies firewalling.
  • UI: Numerous UI improvements, especially wrt Networking: Better handling of multiple NICs per server, with multiple IPv4 and IPv6 addresses per NIC. Introduce distinct "IP addresses" view. The "Create New Machine" wizard has also been extended to support on-the-fly networking configuration of a new machine.
  • UI: Introduce distinct "SSH key" view.
  • UI: Make Google fonts base URL configurable, to support running in networks isolated from the public Internet.
  • UI: Support consistent display of pending actions, e.g., "Shutting Down...", or "Destroying...". Previously, they were reset upon page reload.
  • UI: Support arbitrary groupings of public networks, based on their name.
  • Quota: Support quota enforcement through new 'enforce-resource-cyclades' management command with fine-grained, per-resource actions, to bring current per-user resource usage within specified limits.
  • Statistics: Export basic statistics at the /admin/stats/detail API endpoint. Also make them available at the CLI.
  • Administration: Support mixed DRBD/Archipelago-based Ganeti backends. Previously, a Ganeti cluster was defined as Archipelago-only explicitly.
 Pithos
  • API: Numerous bugfixes and improvements across the codebase. Improve handling of various corner cases, with better error reporting.
  • UI: Support separate view/unsafe domain, for increased security. Users may download their files from a specific view domain, distinct from the API domain, to prevent malicious user content from accessing cookies and other sensitive data set at the level of the API domain. This uses newly-introduced support for OAuth2 tokens in Astakos.
Astakos
  • Projects: Improved project handling: Better logging of project, application, and membership actions.
  • Account API: The Account API has been extended to allow full programmatic handling of projects.
  • Authentication: Remove obsolete /account/v1.0/authenticate call in favor of the standard POST /identity/v2.0/tokens Keystone API call.
  • Authentication: Support OAuth 2.0 Authorization Code Grant, generate and validate OAuth 2.0 access token. This is used by Pithos to support distinct API and view/unsafe domains.
  • Authentication: Support arbitrary attributes as unique identifiers, e.g., eduPersonTargetedId, or ePPN, by extracting them from the ``REMOTE_USER`` header. Also fill third-party signup form fields automatically.
  • Administration: Introduce numerous new admin commands at the CLI, simplify resource and user management commands.
  • Resources: Finer-grained resource control, with individual setting of resource visibility in the API, the UI, and Projects.
  • Statistics: Export basic statistics at the /admin/stats/detail API endpoint. Also make them available at the CLI.
Synnefo-wide
  • Burnin tests: Significant improvements to the Synnefo Continuous Integration mechanism (snf-burnin). It has been extended to also test the Pithos storage service.
  • Administration: Better error logging, and reporting across Synnefo. Admins are notified by default, via email, whenever an unhandled exception occurs in a Synnefo worker.
  • Administration: Ganeti nodes no longer access the Pithos DB at all. Previously, read-only access was required to support spawning VMs from Images on Pithos. As of v0.15, Ganeti nodes only need read-only access to the Pithos block store (NFS or RADOS).
Upgrading to v0.15 is definitely recommended. You can find the corresponding upgrade notes here.

As always feedback is highly appreciated.

Enjoy,
on behalf of the Synnefo team,
Constantinos

Wednesday, February 26, 2014

Synnefo Architecture

Hello everybody,

On a previous post, almost a year ago, we described the services and REST APIs of Synnefo.

Today, we have 2 new diagrams that show Synnefo's layered approach and overall architecture in more detail. First, we extended the diagram from the previous post that presented the zoom-out view, to include all layers from the hypervisor up to the clients (Web UI or command line). We also updated the API layer to reflect the complete OpenStack compatibility that has been included in the latest version. We have now fully integrated Neutron, Cinder and Keystone along with the previously implemented Nova, Glance and Swift:


The second diagram is created from scratch and zooms in to unveil the whole internal Synnefo architecture, showing all Synnefo components and the interconnections between them:


Note that in the left part (Compute/Network) the cloud layer (Synnefo) interacts with the cluster layer (Ganeti clusters) only via HTTP in the effect path and only via AMQP in the update path. The cluster layer does not access any DB, allowing us to scale linearly by adding new nodes on a Ganeti cluster or new Ganeti clusters altogether, even in geographically distinct locations.

Also note, that with Archipelago one can operate over multiple storage technologies
transparently, without the need to change anything on the upper layers, may these be Ganeti or Pithos.

Just to give you a preview on what's coming next, along with many new features, starting v0.16, Pithos will become a thin layer on top of Archipelago or in other words Archipelago will become the backend for Pithos. This will allow for 2 things:
  • Complete unification of cloud storage as seen by the upper layers
  • Complete abstraction of the actual data store underneath
So, hope you find the above interesting and hope the diagrams help everybody to understand the Synnefo architecture a little better.

Thanks,
Constantinos

Monday, January 27, 2014

Synnefo @ FOSDEM 2014

Hello everybody,

the schedule for the Virtualization and IaaS devroom of FOSDEM 2014 has been announced and we are more than happy to be a part of it. After last year's introduction, this year we will be talking about advanced storage integration on Synnefo with Archipelago and Ceph. Check out the talk abstract here.

Of course, we will be making a quick intro to the stack for those who haven't heard about it before, and also do a live demo.

So, join us at the talk, we'll be happy to meet you.

See you all in Brussels.

Tuesday, November 26, 2013

Synnefo v0.14.10 Released

Hello everybody,

we are pleased to announce that today we released Synnefo v0.14.10.

You can find the Debian packages on our apt repository (apt.dev.grnet.gr) under wheezy.

You can also check out the upgrade notes here:
http://www.synnefo.org/docs/synnefo/latest/upgrade/upgrade-0.14.10.html

As you may already know from a previous email on the list:
https://groups.google.com/forum/#!topic/synnefo/rjUMyZTJmWU

Synnefo v0.14.10 is the second transitional package that will help you to smoothly upgrade to Debian Wheezy. Synnefo v0.14.0 is not compatible with Ganeti 2.6 any more, so you will also need Ganeti 2.8 to proceed. You can find the corresponding Ganeti package (snf-ganeti) on our apt repository too. The version you need is:

2.8.2+snapshot1+b64v1+hotplug3+ippoolfix+rapifix+netxen+lockfix2-1~wheezy

The patches: hotplug3, ippoolfix, rapifix and netxen are already merged in the official upstream and will be part of Ganeti 2.10. So, we just backported them in this package. The patches: b64v1, lockfix2 are two small patches that fix some minor issues. The patch: snapshot1 implements some new functionality regarding snapshots which is not used currently by Synnefo v0.14.0, so the codepath is actually inactive, but is there to get stress tested in our production deployment.

So, go ahead, give them a try and please report back any problems or bugs you may find.

Enjoy,
from behalf of the Synnefo team,
Constantinos

Monday, October 7, 2013

Archipelago: officially open source

Hello everybody,

We are pleased to announce that today we are releasing our custom storage layer, Archipelago, as open source software under a 2-clause BSD license. Archipelago has been running in production for over half a year without problems, so after a substantial cleanup we decided to open it up to the public.

Archipelago is a distributed storage layer that decouples Volume and File operations/logic from the actual underlying storage technology, used to store data. It provides a unified way to provision, handle and present Volumes and Files independently of the storage backend. It also implements thin clones, snapshots, and deduplication, and has pluggable drivers for different backend storage technologies. It was primarily designed to solve problems that arise on large scale cloud environments. Archipelago is written in C.

Please check out the official documentation:

You can try it out for yourself, following the instructions found here:

You can also find the code here:
https://code.grnet.gr/git/archipelago

Debian packages can be found on our apt repository under unstable. Add the following to your sources to use them:

deb http://apt.dev.grnet.gr unstable/
deb-src http://apt.dev.grnet.gr unstable/

Finally, on the apt repository, we provide an Archipelago ExtStorage provider, for those that want to use Archipelago with Ganeti.


We hope Archipelago is going to prove useful to others as it did to us.

As always, for comments, questions, or bug reports feel free to contact us at:
synnefo@googlegroups.com

Enjoy,
the Synnefo team

Wednesday, October 2, 2013

Synnefo @ USENIX ;login: (Oct '13 issue)

Hello everybody,

The October issue of USENIX ;login: is out!

And it features an article we've written about Synnefo. Good news is that it is open for everybody, not only USENIX members. So, go ahead and check it out. It is entitled:

"Synnefo: A Complete Cloud Stack over Ganeti"


Enjoy,
Constantinos

Thursday, September 12, 2013

Synnefo @ Ceph Day London

Hello everybody,

The schedule for Ceph Day London has just been announced!

What's more, we will be presenting Synnefo at the event showing how we manage to unify cloud storage (Files, Images, VM disks) using

Synnefo + Ganeti + Archipelago + RADOS in production.

We are also really excited to meet the guys from Inktank and other Ceph users, learn and exchange opinions.

So, if you are in London and the above sound exciting, please join us at the event.

See you in London.