Re: [robocup-rescue-s] 2007 Competition Comments (until now)

From: Hossein Azizpour <azizpour_at_gmail.com>
Date: Sat, 7 Jul 2007 03:47:49 +0330

On 7/7/07, Cameron Skinner <cam_at_cs.auckland.ac.nz> wrote:
>
> Hi everyone,
>
> > We had 11 random maps out of 24 maps which is used for preliminary,
> > which is about half of all maps, I want to know what is the purpose of
> > such a map designing?!
>
> There are several reasons to use random maps.
>
> 1) To prevent hard-coding. Several teams are known to hard-code their
> behaviour for the 4 known maps. This is against the spirit of the
> competition - your code is supposed to work on arbitrary maps, not to be
> optimised for one map in particular.

 I agree with the first part as I had mentioned in my e-mail, to prevent
hard-coding it is a good choice of using Random maps, as I have not
complained in previous competitions, but there is a matter of using learning
methods on complex multiagent systems not optimization, why do you think
there is too many success on 2D Soccer simulation field on AI methods?
I have n't seen many considerable results even on TDPs if you use such
methods, By this fact I do believe that random maps works better with some
if-then-else only codes.
However, just for the sake of information the only data and information
which we have map-specific is the precomputation which we have saved them in
a file to decrease the precomputation time.

2) We simply do not have enough real maps to make an interesting
> competition without random maps.

I think we should create real maps if we do not have enough resources, you
can simply generate 50 random 5 months ago (not two days before competition)
and send them to all teams, then I believe no one can hard code on them, but
anyone can run their agents on the maps in order to be learned.

4 maps over 6 days will get boring very
> quickly.

I think it is not a football match or something and it has no
non-participator spectators to make it not boring for them, come on, we are
not watching movies on projectors, to be boring for us!

If you want to make variety, do not design almost completely identical maps,
just by changing some blockades, or reducing number of fire brigades, have
you ever noticed the difference between the scores of the top 2 teams in
different maps? You think why? as an example today's Kobe-3 and Kobe-4 was
just different in blockades, there are some differences in two teams scores
I will mention it why at the end of my e-mail.
We had too many almost-same maps in previous days.

More importantly, it is practically impossible to add new
> challenges to the existing maps because they have been used so much in
> the past. We must have some way of getting more maps and currently the
> random map generator is the only option.

I just point you to the maps of IranOpen which was held three month ago,
please check the maps, logs and the difference between scores. I believe
there was challenges in the same maps but different scenarios.

You may not have noticed, but today's random maps were a different style
> to those from last year. This is because I wrote new tools for map
> generation two days ago. Today I extended these tools to make tomorrows'
> maps even more different.

So there is, in fact, *more* variety this year than in previous years.
>
> > first is we want a simulation which is most close to the real
> > environment, for overcoming the crisis after an earthquake, if we
> > consider so, which controlling groups/systems knows nothing before the
> > first minute of a disaster and begins its work just after 3 minutes of
> > knowing the city they are working on!
> > Another view is that it should be a test bed for applying methods, which
> > helps to produce/improve some AI/CS methods, in this case, I think
> > current definition and evaluation system of RCRSS has enough randomness
> > (e.g. finding civilians about to die in first cycles, finding easy to
> > extiguish fire sources, ...) if we add to them a random gis in this
> > large scale I want to know what is to be tested on these methods?!
>
> Both views are valid. For the first view, random maps are still useful
> because your agents should work on arbitrary situations, not just known
> ones. For example, when there is a major disaster rescue teams from all
> over the world are called in to help. In the Bam earthquake, rescuers
> from outside Iran could not be expected to know the city very well, so
> for them the map was "random".

Yes of course, but I don't think in every 24 earthquakes in the world we
have 11 helps from outside the country (in the first minute!), I think the
major work and concentrate should be on your own disastrous regions, which
you are responsible of.

The same reasoning applies to the second view. Your methods should work
> on maps with different types of features, and generating random maps
> provides this testbed.
>
> > The other point is why a random map should not have any limitations? let
> > me explain it more by an example of today's first random map(Random 7)
> > on our team. We have worked on an estimation method which predicts the
> > time of traversing a specific route, me and my teammates had a long
> > discussion on maximum size of a "shortest-path" route, the best we could
> > do to avoid the memory exceeding was to find the maximum possible
> > shortest-path route on all the random and non-random maps and scaling it
> > as large as possible, in the Random-7 map our estimator faced with a
> > strange map which has two many disconnections in routes so that a
> > shortest-path route from a corner to the opposite corner contains most
> > of the nodes in the map.
>
> Random-7 had a city split into two halves with a single road connecting
> them. This is very much like many cities around the world, such as
> Auckland, Sydney, San Francisco and others that are built around
> harbours with bridges across them. If your path-planner does not work on
> this type of map then you need to rethink your approach.

It is not problem of path planning, it was the problem length of path which
we have predicted for our estimator, I just said it as an example, yes I did
underestimate it, but as I said just simply add 20000 buildings on a map, it
is a new feature, someone may handle because they do less computations or
whatever, do you call it a novel map to test all the aspects of the codes?

I also note that the shortest path from opposite corners in Random-7
> does *not* contain most of the nodes in the map. In fact, the shortest
> path from opposite corners would be very similar even if the map had
> more connections across the central gap.
>
> > Again, I know it is not against the rules but by the current rules I can
> > design a random map which cause teams' codes to got so many different
> > errors or at least -if they have handled the errors- most of their
> > methods would not be tested.
>
> The goal of random maps is not to break agent code but to provide novel
> maps that test *all* aspects of the competition, including path
> planning. Some maps test the fire brigades more than anything else; some
> maps test the police; some test the ambulances; some test overall
> functions (e.g. path-planning).
>
> There is a difference between errors and difficult maps. Errors, such as
> agents crashing on the test map I sent yesterday, are a problem with
> team code, not the map. Random-7 is a difficult map (at least for path
> planning) but it is not impossible. For example, in 2004 my team had a
> path planner that would work perfectly well on Random-7 because it
> worked by splitting the road network into connected subgraphs. Don't
> blame the map if you have a poor path planner.

Again, that was not the problem of path planner in our code.

> However, I appreciate your efforts on RCRS while it has no benefits for
> > you personally, but I think these In-My-Opinion careless decisions is
> > disappointing everyone about this field in Robocup event.
>
> These decisions are not careless. A lot of thought goes into coming up
> with ways to make the competition more interesting and challenging, and
> design decisions are made for a reason. As I said before, you are most
> welcome to contribute maps of your own, or tools for generating new
> maps. Perhaps you could implement a module for the random map generator,
> or extend the scenario maker, or write your own map generator?

I think it is, I think the important thing is to make a fair competition
more than a interesting competition(I don't what is the definition of
interesting in RCRS) I have sent an e-mail to the mailing list three days
that there are difference in the same situations with same randomseeds, you
said you will investigate it and announce it as soon as possible, I didn't
see any e-mail in the mailing list, as your information today's maps has the
same problem too, on Kobe-3 Impossibles and Hinumiyogara are different from
others, in Kobe-4 Impossibles, Naito Rescue and Roboakut are different from
others.
Which one is more important!

As I remember the most useful tool for creating scenarios is the one that
S.O.S. team has implemented (by previous leader/member Arash Rahimi).

Regards,

> Cameron.
>

-- 
Hossein Azizpour,
Robotics Research Center,
Amirkabir University of Technology,

_______________________________________________
robocup-rescue-s mailing list
robocup-rescue-s_at_cc.gatech.edu
https://lists.cc.gatech.edu/mailman/listinfo/robocup-rescue-s
Received on Sat 07 Jul 2007 - 00:21:53 GMT

This archive was generated by hypermail 2.2.0 : Sat 07 Jul 2007 - 00:21:54 GMT