Congratulations RTEMS GSoC 2013 Students

Congratulations to the accepted students in GSoC 2013 for the RTEMS Project! We had many fine applicants again this year, and were able to accept 9 students to do projects with us this summer. In alphabetical order by project name, we accepted these students/projects:

• Shubham Somani
• Application Configuration GUI for RTEMS.
• Deng Hengyi
• Atomic Operations and SMP lock debug tool for RTEMS
• Dhananjay M Balan
• Better debugging support for RTEMS in GDB.
• Hesham Moustafa AL-matary
• Enhance low-level API of libmm (Memory Protection & Caches)
• Philipp E
• Paravirtualization layer in RTEMS
• Jin Yang
• Porting CAN driver, LinCAN, to RTEMS
• Peng Fan
• RTEMS Runtime Loader
• Sree Harsha Konduri
• SMP Aware Scheduler
• Vipul Nayyar
• Unified APIs

These are all important, ambitious projects that will be of great benefit to RTEMS if successful, and will definitely teach the students a lot about project management and open source development.

Software Licenses with RTEMS

The RTEMS license is a modified version of the GPL version 2 that includes an exception to permit including headers and linking against RTEMS object files statically. Normally, the GPL can only be linked statically with other GPL code, or rather, linking statically with GPL code would cause your code to become GPL code. The LGPL is not a suitable alternative, because it either requires use of a shared library that can be re-linked, or release of the linked (application) code. And newer versions (GPL version 3) are completely unsuitable for embedded systems due to the relinking restriction which is technically challenging.

A problem for RTEMS is there are no copyleft licenses that are compatible with the RTEMS license. Thus, RTEMS Project has to reject any code that uses the GPL or LGPL, even though RTEMS seems to use the GPL itself---this is because of the exception for static linking, and also because an upstream GPL version 2 project could at any time switch to GPL version 3 and become totally unusable. In practice, RTEMS can only accept original code contributed under the RTEMS License and code that has a permissive license.

I could not find any license that provides the copyleft protection of a software project while still allowing static linking of proprietary software. Maybe there is some subtle legal or technical issue that I do not understand, but it seems like such a license ought to exist somewhere that protects the free software while permitting applications to use it; a sort-of "Embedded GPL".

Some things that RTEMS could do better include:

• Collect all of the copyright and license disclaimers for users
• Collect all of the advertising restrictions, or move those encumbered files to a secondary repository
• Switch from the GPL + linking exception, but to what I do not know

Update 5/8/13: Identified that RTEMS uses version 2 of the GPL, and give some background on why RTEMS has not and will never switch to version 3.

RTEMS on gem5 SPARC

I previously wrote about booting RTEMS on M5 (Now gem5) SPARC_FS. I am following up on this work in preparation for the release of RTEMS 4.11, which will hopefully be coming soon. I compiled RTEMS (with two small patches) and booted two samples on gem5 and on Simics Niagara.

On the simulator side, the instructions I gave previously continue to work for booting both OpenSolaris and RTEMS. I updated some of the links in the prior post to reflect the change in host to Oracle for the OpenSPARC tools. One issue I did not notice at first is that the port for the console to connect m5term to the gem5 simulator is port 3457.

I successfully built and booted hello and ticker with the niagara BSP. There are a few patches that need to be applied to RTEMS, but hopefully I can get those patches committed before releasing 4.11. Ideally, the RTEMS 4.11 will be able to compile and boot with gem5, giving the RTEMS community an open-source simulator for testing.

Interestingly, the ticker appears to count time. I'll have to take a look at how gem5 simulates the (s)tick register. Simics does not simulate the timer with any accuracy.

I think Qemu would be a nice target simulator for some future endeavors, but I do not have the time to investigate the feasibility of running RTEMS on the Qemu Sparc64.

Critical Bugs and Quality Assurance

Sebastian Huber recently posted a nasty RTEMS bug and fix. While simple, the bug manifested in their application as an increase in one task's latency from 20us to 170us! The cause of the problem was that RTEMS has two kinds of critical sections—dispatch and interrupt—and new SMP-aware code added a dispatch critical section to the thread dispatch code. The problem happens with overlapping a traditional interrupt disable critical section. The SMP dispatch code looks like:

void _Thread_Dispatch(void) {
  Thread_Disable_dispatch();
  SMP_Dispatch_other cores();
  Interrupt_Disable();
  ...
  ... do things, including context switch if needed
  ...
  Interrupt_Enable();
  // problem here!
  Thread_Unnest_dispatch(); // enable
  ...
}

The problem is that an interrupt can occur between Interrupt_Enable and Thread_Unnest_dispatch. Suppose a low-priority task (L) is executing _Thread_Dispatch, and the interrupt enables a high-priority task (H), but
H will not be dispatched because dispatching is disabled. Instead L enables dispatching and resumes executing, which is a priority inversion!

The fix reverts the changes made for SMP. For the SMP code, the priority inversion still exists and is unresolved. (RTEMS currently does not make real-time guarantees for the SMP support, so no one cares yet.)

In the broader picture, the bug seems like it should be easy to detect. The issue with free open-source software (FOSS) is that quality assurance (QA) is almost non-existent: the "many eyeballs" philosophy argues against QA. But what else can FOSS do? No one is going to pay for extensive testing, and if they do they have no incentive to share.

FOSS communities (and corporate developers) need better tools for software QA. This summer RTEMS had a GSOC student who was looking at testing. Testing is probably the first tool in the QA toolbox, and the only one most developers have a clue about; how about static analysis, path coverage, standards conformance, or certification? Some interesting work modeling, proving, and certifying systems is out there: Where is the undergraduate textbook and course on QA?

GSOC2012: MMU project and musings

I finally made a pass at merging my student's final code for his GSOC2012 project. The project was the culmination of multiple summer projects and some work I did. I'm excited to try improving it to add sparc64 support and use it in my research.

While revising the student's submission for reviewing and merging, I thought of a few ways to improve future new developer participation:

• Better github integration. Staying up-to-date with rtems.git, tracking student progress, and getting code review from more developers would be helpful for quicker turnaround on submissions.
• Style: documentation and code conventions. Clear, consistent guidelines and examples of proper/improper coding style would make reviewing and merging a lot easier.
• Improved Git Workflow. Teaching students how to make useful branches and commits ahead of time would ease code merging, testing, and revising.
• More submissions. We need to get code reviewed if not merged in smaller increments; this need is well-known and repeated.

These improvements could be addressed in part when students/developers are new to the scene. For example, instead of just making students prove they can build RTEMS and patch hello world, we could require them to document and fix the style of sample code using a branch, and submit a pull request for RTEMS github that contains their proof as separate commits on the same branch.

For getting students to submit and be reviewed more often will take more work on behalf of mentors, developers, and students. Something that may help would be requiring code review as part of the weekly status meetings we instituted this GSOC. Perhaps each student's weekly commits can be reviewed by their mentors as part of tracking progress and status.

Institutional support from RTEMS mentors and developers would help. For example, github integration requires developers and mentors to use github. Style consistency requires a style guide that we accordingly maintain and abide by. Teaching good workflow, and fixing the bad, takes effort: mentors need to (know how to) identify and correct a student who struggles. Increased submission frequency requires urging and commitment from mentors to review code. These improvements take effort, but I think they could substantially improve the participation, progress, and production of students.

Crufty Stuff

Joel Sherrill posted back in 2010 about the lines of code in RTEMS. I was curious about something related to this so first I replicated his work using sloccount. I think he may have neglected the inline files (because I had to add the .inl extension). I also added support for m4, .ac, and .am files. The latter two are part of the autotools build system and are filed under 'makefile'. Here are the current results on the RTEMS tree

Totals grouped by language (dominant language first):
ansic:       745153 (86.51%)
asm:          38574 (4.48%)
makefile:     34934 (4.06%)
ada:          26261 (3.05%)
sh:            7086 (0.82%)
cpp:           5248 (0.61%)
m4:            2996 (0.35%)
pascal:         814 (0.09%)
perl:           279 (0.03%)

Total SLOC  861,345

Now for the interesting part: I also ran sloccount after running bootstrap, which generates a bunch of files for the build system. I modified sloccount to also count the Makefile.in files (which are generated with autotools from Makefile.am sources). Here are the results

ansic:       745153 (35.73%)
sh:          638471 (30.61%)
makefile:    557257 (26.72%)
m4:           73475 (3.52%)
asm:          38574 (1.85%)
ada:          26261 (1.26%)
cpp:           5248 (0.25%)
pascal:         814 (0.04%)
perl:           279 (0.01%)

SLOC*     2,085,532
* includes generated files

And do you know how many humans look at that generated code? Any idea how much of it is tested?

It would be interesting to include all the other code that goes into making RTEMS work such as the compiler, RPM packing scripts, and so on. The moral of this data is that the build system is complex. Very complex.

This data was generated using David A. Wheeler's 'SLOCCount'.

Using partitions as a free list

Yesterday I discussed my search for some help with implementing a free list in RTEMS. Today I'm glad to say I was able to use partitions to achieve my objective of a growable free list—something akin to a slab allocator. A fundamental limit, the maximum number of partitions, still exists but partitions can have an arbitrary size; these limits might be circumvented safely by making each partition much larger than the previous or by combining old partitions into new ones. Further improvements can be made, but for now I'm satisfied.

Allocation
Iterate over available partitions until one returns a free object. If none succeeds, create a new partition (malloc a new set of objects) and get a free object from the new partition.

I have not handled what to do if partition creation fails. Performance could be improved by tracking the non-full partitions.

Deallocation
Return the object to its partition. I made this easy by storing a handle to the owning partition as part of each object. Non-invasive techniques, such as tracking address ranges for each partition, could be used for a generic solution.

RTEMS free list?

I implemented a free list in an RTEMS application as a pre-allocated array of objects on a linked list, but I foresee the need to have a fail-over case for growing my free list dynamically. Extending my current implementation to support dynamically changing the size of the free list is easy enough, but I'm interested to see if such support exists. I found two likely candidates: Partitions and Regions.

Partitions implement a simple free list of objects and have low overhead in both time and space. However a partition cannot change the size of its memory so an allocation will fail after the partition exhausts the preallocated set of buffers. This behavior is exactly what I already have implemented for my application.

Regions implement a more complex memory manager that allows for carving up a chunk of memory, calling it a region, and allocating segments from within a given region. However since segments can be variable-sized, the region manager uses a first-fit allocator that coalesces free blocks. Such an allocator is overkill for a free list of fixed-size objects.

Neither are quite what I want; where is the middle ground. My suspicion is that no such mechanism exists already because (hard) real-time applications usually have well-known resource requirements; the number of objects in the system are known and the partition manager would be sufficient. But when an unknown number of same-sized objects are employed, the current mechanisms appear to be insufficient.

Interestingly the Object Manager, an internal RTEMS subsystem, does seem to have capabilities to support dynamically extending the size (up to a preconfigured but possibly unlimited maximum) of a free list of fixed-size objects. Unfortunately the Object Manager is not accessible directly from the application level without violating API visibility rules.

Update: See my follow-up post regarding using partitions to implement a growable free list.

RTEMS Memory Protection Middle Layer

This is a continuation of my prior post in which I described an API for memory protection. Here I will describe the middle layer that glues the user API to the CPU support code that eventually interacts with hardware to enforce the permission attributes.

For those familiar with RTEMS, this code lives in libcpu and the design mimics that of the cache manager. The middle layer is just 5 functions:

/*
 * Initialize the hardware to prepare for
 * memory protection directives.
 */
rtems_status_code _CPU_Memory_protection_Initialize( void );

/*
 * Make sure memory protection permission
 * is valid for this CPU
 */
rtems_status_code _CPU_Memory_protection_Verify_permission(
    rtems_memory_protection_permission permission
);

/*
 * Check if memory protection region size
 * is valid for this CPU
 */
rtems_status_code _CPU_Memory_protection_Verify_size(
    size_t size
);

/*
 * Install (enforce) the memory protection entry mpe
 */
rtems_status_code _CPU_Memory_protection_Install_MPE(
    rtems_memory_protection_entry *mpe
);

/*
 * Uninstall the memory protection entry mpe
 */
rtems_status_code _CPU_Memory_protection_Uninstall_MPE(
    rtems_memory_protection_entry *mpe
);

These functions are called via thin wrappers from the high-level API. The install and uninstall entry functions are called iteratively by the higher-level functions that install and uninstall protection domains. The two "verify" functions are used by the high-level API to avoid trying to install invalid entries proactively. The remaining functions interact with the hardware directly to support the memory protection mechanisms.

Next I will briefly explain what each function does in my sparc64 implementation.

• _CPU_Memory_protection_Initialize
• Disables interrupts
• Selectively demaps pages that were loaded during system startup
• Flushes the cache
• Maps new pages for the kernel core services
• Installs exception handlers for MMU misses
• _CPU_Memory_protection_Verify_permission
• Returns true; needs a little refinement.
• _CPU_Memory_protection_Verify_size
• Ensures the size (bounds) is a multiple of the supported TLB page size
• _CPU_Memory_protection_Install_MPE
• Adds a TLB entry for the memory region specified in the MPE
• _CPU_Memory_protection_Uninstall_MPE
• Removes the TLB entry for the specified memory region

These functions call BSP- and CPU-specific functions to handle the hardware (TLB) interactions and comprise the entire middle layer between the API for memory protection and whatever low-level code enforces the protection mechanism.

RTEMS Memory Protection API

I am working on a solution to introduce real-time memory protection to RTEMS. I started from the work of two GSOC projects related to MMU support in RTEMS, and you can see most of the raw code that I have at http://code.google.com/p/gsoc2011-rtems-mmu-support-project/. This post describes the current design of the (high-level) API layer for setting attributes on regions of memory. I plan to post in the future about the middle and low-level interfaces. Any suggestions about design/naming/implementation etc. are welcomed.

The high-level interface comprises a region and attributes to create a memory protection entry (MPE), a set of which constitute a memory protection domain (MPD). Three structures are visible at the API layer, although only regions should ever be directly accessed; MPEs and MPDs are only meant to be used as handles in userland.

/**
 * A region of contiguous memory
 */
typedef struct
{
 char  *name;
 void  *base;
 size_t bounds;
} rtems_memory_protection_region_descriptor;

typedef uint8_t rtems_memory_protection_permission;

/**
 * A memory protection entry (MPE) is a region of contiguous memory
 * (rtems_memory_protection_region_descriptor) with a set of permissions.
 * If it is currently being enforced then the installed field is true.
 */
typedef struct
{
 rtems_chain_node                          node;
 rtems_memory_protection_region_descriptor region;
 bool                                      installed; 
 rtems_memory_protection_permission        permissions;
} rtems_memory_protection_entry;

/**
 * A memory protection domain comprises a chain of active MPEs and 
 * a freelist of idle MPEs. MPE storage is allocated from the Workspace
 * and managed in the MPD.
 */
typedef struct
{
  rtems_memory_protection_entry  *mpe_array;
  size_t                          mpe_array_size;
  rtems_chain_control             active_mpe_chain;
  rtems_chain_control             idle_mpe_chain;    /* free list */
} rtems_memory_protection_domain;

Application developers can create regions, for example

rtems_memory_protection_region_descriptor text_region = {
    .name = "text",
    .base = TEXT_BEGIN,
    .bounds = TEXT_SIZE
  };

Where TEXT_BEGIN and TEXT_SIZE are specified somewhere. The advantage of using a name to identify a region is that eventually we could layer a filesystem over the MPD structure and open memory files with specified attributes, for example: int fd = open("/memory/text", O_RDWR);. I have not given thought to how best to create regions, but that interface should be orthogonal to the protection API.

Regions must be encapsulated within MPEs and added to a MPD to have permissions enforced.  The requisite functions are

/**
 *  Allocates an array of max_number_of_mpes 
 *  memory protection entries from the workspace.
 */
rtems_status_code rtems_memory_protection_initialize_domain(
  rtems_memory_protection_domain *domain,
  size_t max_number_of_mpes
);

/**
 * Create a new memory protection entry for the region with
 * permissions in perms. This function adds the newly created mpe
 * to the active chain of the_domain but does not install it. 
 */
rtems_status_code rtems_memory_protection_create_entry(
  rtems_memory_protection_domain* the_domain,
  rtems_memory_protection_region_descriptor * const region,
  const rtems_memory_protection_permission perms,
  rtems_memory_protection_entry** mpe_ret
);

/**
 * Install all mpes on the active list of the_domain.
 */
rtems_status_code rtems_memory_protection_install_domain(
  rtems_memory_protection_domain* the_domain
);

So to enforce the text_region from above:

rtems_memory_protection_domain *protection_domain;
rtems_memory_protection_entry *mp_entry;
rtems_memory_protection_permission permission;

rtems_memory_protection_initialize_domain( protection_domain, 8 );

permission = RTEMS_MEMORY_PROTECTION_READ_PERMISSION | 
             RTEMS_MEMORY_PROTECTION_EXECUTE_PERMISSION;

rtems_memory_protection_create_entry(
    protection_domain,
    &text_region,
    permission,
    &mp_entry
);

rtems_memory_protection_install_domain(protection_domain);

One aspect I'm not happy with is the permissions field, an 8-bit field with preprocessor macros defining masks. A better solution would improve usability. The remainder of the API layer are a handful of functions to manage MPEs (find mpe by address/MPD, delete mpe from mpd, set permission) and MPDs (uninstall all MPEs and finalize to free resources).

Update: continued here

GSOC Mentor Summit

This summer I mentored for the RTEMS project as part of the Google Summer of Code (GSoC) and was invited to attend the mentor summit, which is an unconference with mentors from the projects that participated in GSoC. Bringing together representatives from across open source communities is one of the benefits of the GSoC program: Many shared interests exist among the different organizations that participate in GSoC, and the social interactions and networking opportunities are nice.

I organized a session on open source in space, and I also got a lot out of the sessions on Women and new contributor outreach discussion, Non-Profit Infrastructure for Software Freedom, the Open Source OS Summit, and Aging Project Infrastructure Evaluation. The latter two were technical whereas the session on space focused on raising awareness of the contributions to space science that FOSS makes. The session on contributor outreach gave advice on how to be more inclusive and welcoming to outsiders (including women, who are underrepresented in FOSS communities), and the non-profit session discussed the why and how of making an open source software project benefit from charitable donations.

Spacecraft Flight Software Workshop

MMS: a NASA mission
that will fly RTEMS

Last week I attended the Workshop on Spacecraft Flight Software (FSW 2011) at the Johns Hopkins University Applied Physics Laboratory (APL). This workshop brings together engineers from throughout the space flight software community especially those from GSFC, APL, and JPL. Joel Sherrill had more to say about this on his blog.

As one of the few RTOSs that run on radiation-hardened platforms, including the LEON and RAD750, RTEMS enjoys a share of space missions both in the US and abroad. Among the trends at FSW 2011 was an increase in the number of RTEMS-related talks over prior years: Open source does well in a recession.

I got to see a little of how space users are using RTEMS, what they care about, and where development should proceed. Besides V&V, which are important to the FSW community, the trends in RTOS use appear to be toward encouraging reusable software components (mainly through OSAL), flexible runtime support for dynamic execution such as scripting and dynamic linked code, memory protection, improved storage, autonomy, and simulation. Among these, RTEMS will soon be supporting dynamic code loading thanks to Chris Johns, and I am investigating memory protection. With respect to storage, one of the talks described a FAT32 flash filesystem for VxWorks and RTEMS. Autonomy is important for deep-space missions in which communication with ground support is delayed (or obstructed) thus preventing real-time interaction.

All-in-all I thoroughly enjoyed the workshop for its technical and social aspects. I met a lot of smart, kind people who work hard to advance space science and exploration.

multilib

Back when Eugen and I were porting SPARC-V9 to RTEMS, we had to consider the compiler tools for cross-compiling RTEMS to the target architecture. One of the issues was that RTEMS uses the multilib feature of GCC for building architectural variants, so we should consider the multilib options for SPARC-V9 and submit patches to add support for the desired multilibs to the rtems-sparc64-gcc.

What are multilibs?
The GCC manual doesn't say anything other than to identify the flags that use multilib. Google isn't terribly helpful here either. The automake manual has this to say:

18.3 Support for Multilibs

Automake has support for an obscure feature called multilibs. A multilib is a library that is built for multiple different ABIs at a single time; each time the library is built with a different target flag combination. This is only useful when the library is intended to be cross-compiled, and it is almost exclusively used for compiler support libraries.

The multilib support is still experimental. Only use it if you are familiar with multilibs and can debug problems you might encounter.

Hardly helpful.

So we ignored multilibs and moved on. That was fine, although to this day the sparc64-rtems-gcc does not support multilibs. If a tighter integration of the sparc64-rtems-gcc and BSPs becomes important then this issue might be worth re-visiting. But I still had little knowledge about multilibs.

multilibs are...
It turns out that multilib is common, although most end users won't notice them. multilib is the process by which a library is built multiple times, each with a different set of compiler flags. That means a library can be linked to applications built with similar flags, which may be necessary in case the compiler flags change the ABI (application binary interface). An example is compiling legacy 32-bit applications on 64-bit machines: current Linux distros ship GCC with a 32-bit multilib, so the GCC libraries are available for compiling 32-bit applications. On a 64-bit computer without a 32-bit multilib gcc cannot compile 32-bit programs.

RTEMS adds multilib flags to its rtems-flavored gcc compilers to allow RTEMS to link to GCC libraries that best support the target hardware. For users of existing board support packages (BSPs), the multilib is invisible: the BSP specifies compiler flags (in make/custom/somebsp.cfg), and rtems-gcc links the right library. However, BSP developers should ensure that critical flags specified by the BSP match an available rtems-gcc mulitlib or else a new multilib should be contributed. The gcc flags will pull in a multilib if there is a match in gcc-print-multi-lib

On the RTEMS multilib page there is also an explanation of how RTEMS can build its own multilib library, the librtemscpu.a. Building a multilib RTEMS is mostly useful to save a little time and space when compiling many variants of the same processor family, for example multiple BSPs from the same CPU. Perhaps I'll follow up sometime to discuss RTEMS' librtemscpu.a

RTEMS GSoC 2011

Congratulations to the students accepted to participate in the GSoC 2011 with RTEMS as their mentoring organization!

Student: Christophe Huriaux
Project: Implementation of the ISO9660 filesystem
Goal: Implement the ISO9660 filesystem so that RTEMS can read CD, DVD, and similar media.

Student: Ricardo Aguirre Reyes
Project: POSIX Timing Tests
Goal: Implement a suite of tests to time POSIX routines.

Student: Cui Xiang
Project: POSIX Compliance Test Suite (misnomer)
Goal: Implement a suite of tests for filesystem testing.

Student: Petr Benes
Project: Porting of resource reservation framework to RTEMS executive
Goal: Port the FRESCOR resource reservation framework to support RTEMS.

Student: Quanming Shi
Project: RTEMS MMU Context Support
Goal: Implement an architecture independent API and structure to manage memory blocks with access attributes based on MMU Support.

Student: Scotty Smith
Project: Lua Scripting and Shell Support in RTEMS
Goal: Add Lua as a library in RTEMS and provide a Lua-based shell as an alternate to the current shell.

Student: Zhang Wenjie
Project: Hypervisor for RTEMS
Goal: Implement virtualization support in RTEMS kernel and demonstrate Linux running on RTEMS.

Student: Jie Liu
Project: RTEMS port of the GNU Java Compiler
Goal: Support compiling Java applications on RTEMS with the GCJ (for x86 at least).

RTEMS: Adding a new scheduler

In this post I describe how to add a new scheduler implementation to the RTEMS open-source real-time operating system. The scheduler I use as an example is an earliest deadline first (EDF) scheduler which applies the EDF algorithm to schedule periodic tasks and a FIFO policy for non-periodic tasks.  This post demonstrates how to use my modular scheduling framework to extend the scheduling capabilities of RTEMS.

Interrupt Handling with MMU Hardware on an MMU-less OS

This post describes some work that I have just recently gotten to a point that it seems to be correct.  It was a long-running project that started around May 2010.

RTEMS Modular Task Scheduler

As I mentioned in my last post, this past summer I participated in the Google Summer of Code by working on the RTEMS project. I have hopefully ironed out the last few small details that prevented the basic infrastructure that I developed from being accepted into the RTEMS code base. In this post I describe the design of the new scheduler, and also comment on some of the problems that I faced when merging the work. At the end of the post is a list of remaining work for this project.

Pre-existing RTEMS Scheduler

New Modular Scheduler
  Scheduler Control Structure
  Ready Queue Encapsulation
  Per-Thread Metadata
  Scheduler Operations Table
  Configuring the Scheduler

Merging the New Scheduler
  Short-circuit Evaluation
  PowerPC overheads

Open Issues
Improper Encapsulation
Interrupt Service Routine (ISR) Latency
Documentation
Configuration
New Technologies

Retroactive Introduction to my GSOC project

I've been busy this summer, between my GSOC project, my academic progress, and life in general. I thought I should catalog some of the work that I've done for the GSOC 2010. I have previously talked about my work to port RTEMS to SPARC V9. Well, I had a GSOC project accepted for the RTEMS project which I have been working on this summer. The remainder of this blog post is excerpted from my project proposal and describes what I have been doing lately. The full project proposal can be read online at: http://code.google.com/p/rtems-sched/wiki/ModularSuperCoreSchedulerProject.

My project is to refactor the RTEMS task scheduler to make it more modular. The current RTEMS scheduler is tightly coupled with the thread management code. This tight coupling prevents easy implementation and testing of alternate scheduling mechanisms. Refactoring the scheduler into a new SuperCore Scheduler Manager enables RTEMS developers (namely, me) to explore new scheduling mechanisms.

After finishing my project, the RTEMS scheduler internal data structures will be encapsulated in a module that provides a protected interface such that none of the thread management code directly touches scheduling data.

Perhaps the most important outcome of my project will be enabling exploration of scheduling for symmetric multiprocessing (SMP) and multicore platforms. An SMP system is a closely-coupled system with multiple CPU chips interconnected with main memory; a multicore system is even more tightly coupled, with multiple CPUs interconnected on a single chip. Although RTEMS currently supports multiprocessing systems, the support is designed for coarse-grained systems. Each processing element (core) runs a separate RTEMS application and executive, and communication is routed through a portion of shared memory. Scheduling decisions are made locally at each core, and tasks are not able to migrate between cores.

Although my project will not provide SMP support, the goal of refactoring the scheduler is to facilitate implementing an SMP scheduler, so care will be taken to make sure the interface does not enforce any policies that might deter an SMP scheduler. By supporting global scheduling decisions and task migration, RTEMS will be able to balance a processing load across multiple CPUs. Further, unifying the executive across all the cores in the system allows RTEMS to make smarter decisions and to reduce the effort of end users in deploying SMP and multicore configurations.

Building and Booting RTEMS sparc64

I've put together a package at http://code.google.com/p/rtemssparc64/downloads/list called buildsparc64.tgz that contains a directory structure with scripts to help build the niagara and usiii BSPs.

There is a README with some simple instructions. After extracting the archive, you'll need to create a link in the build-sparc64 directory to the RTEMS sources on your machine (that contain the sparc64 CPU model and BSPs). Assuming you have the sparc64-rtems4.11 tools installed, you will then be able to build the niagara or usiii BSPs. Then there are scripts in the boot subdirectory to help build a bootable ISO image for either BSP. The scripts are set up to only build and package some of the RTEMS sample applications. Note that only Niagara has been run on open source simulators so far. I have run both BSPs on the Simics simulator, and you can contact me privately for more information.

Once you successfully build the image.iso file, you can follow the instructions at my previous post about the M5 simulator to run the BSP. Simply make a link from the built image.iso file to the /dist/m5/system/disks/image.iso file, and make the changes I mention to the M5 configuration files.

SPARC V9 (sparc64) RTEMS

I'm pleased to announce that the work I've done, together with my colleague Eugen, to get RTEMS working on the sparc64 processor architecture, has been accepted for inclusion to the RTEMS project. We've been working on porting RTEMS to sparc64 for about 9 months, and have had a functional port for about 4 months.

We started with a basic template from the 32-bit SPARC V7 port of RTEMS, and modified or replaced its code with the appropriate support for the 64-bit SPARC V9. There were quite a few changes, since we started with SPARC V7 and skipped ahead to V9. The SPARC V8 truly is somewhere between the two ports. For user-space compatibility, the processor is not substantially different; however, the privileged processor state is very different in SPARC V9, and even more so in the sun4v variant in which we were most interested. I've shared a Updated: Google Doc that contains a lot of our notes from the porting process. We started by focusing on the SPARC Niagara CPU, but have since added support for the UltraSPARC 3 CPU model and should be able to run most UltraSPARC family processors.

A description of the Board Specific Package (BSP) for the Niagara CPU can be found on the RTEMS Wiki at the http://www.rtems.com/wiki/index.php/Niagara page. I'm still working to get instructions on how to run the BSP in the M5 open source simulator, which I hinted at in my previous post about M5.

It was a fair amount of work to clean up the code to submit it to RTEMS, but it will be nice to have the code in the repository. I guess we'll see if anyone else is actually interested in using it! One nice aspect, and I think one of the reasons the RTEMS people accepted the work, is that there are no other 64-bit architectures currently supported by RTEMS. So adding the SPARC V9 processor family provides a good way to test RTEMS' 64-bit capabilities. Eugen and I did not have too much difficulty in this regard, at least from a functionality point-of-view.

Hopefully now that the port is "upstream", I can again focus on my research efforts actually using RTEMS on the sparc64, and not just getting things to work!